simple dom problem

[someoneneedshelps]

Im search some html for the £4.50 out of this...

<span id="fshippingCost" class="notranslate sh-cst "><span>£4.50</span>

using this code

	foreach($html->find('span[id=fshippingCost]') as $shippingcost)
	{
		$arrayresults[] = $shippingcost->plaintext;
		$shippingcoststr = $shippingcost->plaintext;
	}

strangest result back is £17.33 why???

 

[someoneneedshelps]

So if there is one instance of span in one file and one instance in another why should it fail chris?

 

[ChrisHirst]

and one instance in another why should it fail

I have absolutely no idea, as I have said a few times (in different threads) I prefer to see evidence and leave absolute certainty without evidence for religion. ... So, ... as stated earlier in the thread, to determine why it fails would require access to the original source data, one that succeeds and [ideally] two that fail.


Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[someoneneedshelps]

looks at an source file and the areas are same... heres two pages, I cant list the full page its massive

		<div class="u-flL sh-col">
					<div class="sh-gspFirstLine" style="">No additional import charges on delivery.</div>
					<div class="sh-gspSecondLine " >
						This item will be sent through the Global Shipping Programme and includes international tracking. <a href="[URL unfurl="true"]http://pages.ebay.co.uk/buy/globalshippingprogram/"[/URL] target="_blank">Learn more<b class="hideforacc">- opens in a new window or tab</b></a></div>
					<div class="sh-gspDashedBorder"></div>
					</div>	
	</div>	

						
<div>
	<div class="u-flL lable" id="shippingPlaceHolderId">Postage:</div>
	<div class="u-flL sh-col">
	<span id="shSummary">
		<!-- TODO:check whether to display none or not -->
					<span id="fshippingCost" class="notranslate sh-cst ">
						<span>£2.85</span>
						</span>
				<span id="fShippingSvc">
				Economy Delivery<!-- GSP -->
						</span>
		<span> | </span>
		<span>

this is the other page, both give wrong postage

		<div class="u-flL sh-col">
					<div class="sh-gspFirstLine" style="">No additional import charges on delivery.</div>
					<div class="sh-gspSecondLine " >
						This item will be sent through the Global Shipping Programme and includes international tracking. <a href="[URL unfurl="true"]http://pages.ebay.co.uk/buy/globalshippingprogram/"[/URL] target="_blank">Learn more<b class="hideforacc">- opens in a new window or tab</b></a></div>
					<div class="sh-gspDashedBorder"></div>
					</div>	
	</div>	

						
<div>
	<div class="u-flL lable" id="shippingPlaceHolderId">Postage:</div>
	<div class="u-flL sh-col">
	<span id="shSummary">
		<!-- TODO:check whether to display none or not -->
					<span id="fshippingCost" class="notranslate sh-cst ">
						<span>£4.50</span>
						</span>
				<span id="fShippingSvc">
				Economy Delivery<!-- GSP -->
						</span>
		<span> | </span>
		<span>

 

[ChrisHirst]

heres two pages, I cant list the full page its massive

URLs please, so the whole page can be seen not just the bit that you assume it should only find.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[someoneneedshelps]

http://www.ebay.co.uk/itm/Asics-Gel-3020-Running-Shoes-/182134106150?ssPageName=STRK:MESE:IT

 

[someoneneedshelps]

Not sure if this helps but the original problem, that figure is now £17.35

 

[someoneneedshelps]

seems i may of stumped everyone with this problem, i gone back to this that gets the text but its wrong value

        foreach($html->find('span[id=fshippingCost]') as $shippingcost)
	{
		$shippingcoststr = $shippingcost->plaintext;
	}

 

[ChrisHirst]

Assuming the one URL provided is a working one, how about ones that do not?

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[Olaf Doschke]

Seems to me this is a DOM vs originally loaded HTML issue. The source code browsers show is an initial code, it's not current DOM. Ebay is showing shipping costs depending on your location detected after the pages have loaded. So original value might be for US users only or UK or whatever. Then the browser gets acitve, the shipping costs change and to get the price displayed you'd have to parse the runtime DOM.

With $html = file_get_html($iUrl); you request the original HTML code, you don't get the browsers DOM.No javascript runs and adjusts the DOM to what you see online.

Bye, Olaf.

 

[Olaf Doschke]

The "art" of site scraping has been replaced by using APIs. eBay has an API call for what you want to find out:

http://developer.ebay.com/DevZone/shopping/docs/CallRef/GetShippingCosts.html

Start using APIs, stop scraping websites.

Bye, Olaf.

 

[someoneneedshelps]

Ok, OlafDoschke I get what your saying, am I not viewing and/or scraping the source that is delivered to me? ie: the page above, click view source I thought that is wehat I'm working with, what are the mechanics using ebays api's? example to getshippingcost etc.? is this

http://php.net/manual/en/domdocument.loadhtml.php

pointed out by Chris not better than simpledomparser?

 

[someoneneedshelps]

To use this are fees involved?

$apicall = "$f_endpoint?OPERATION-NAME=findItemsAdvanced"
. "&version=$f_version"
. "&GLOBAL-ID=$globalID"
. "&SECURITY-APPNAME=$appID" // replace this with your AppID
. "&RESPONSE-DATA-FORMAT=$responseEncoding"
. "&itemFilter(0).name=Seller"
. "&itemFilter(0).value=$sellerID"
. "&itemFilter(1).name=ListingType"
. "&itemFilter(1).value=$itemType"
. "&paginationInput.entriesPerPage=$maxEntries"
. "&sortOrder=$itemSort"
. "&affliate.networkId=9" // fill in your information in next 3 lines
. "&affliate.trackingId=123456789"
. "&affliate.customId=456";

 

[someoneneedshelps]

Assuming the one URL provided is a working one, how about ones that do not?

The id of the span element finds what im looking for but last few threads show a different light on the subject

 

[Olaf Doschke]

Browser view source show initial HTML. Once that's loaded into a Browser, it starts processing it, including starting javascripüt functionaliites updateing content here and there.
This is not happening, if you only request the HTML, also if you put it in a DOM model, this is not an active Browser with all its whistles and bells.
So you can only see the initial HTML or it's DOM representation.

Pricing mainly is free, see:

http://developer.ebay.com/join/pricing/

You pay for support only, and you "pay" with a rate limit you can vastly expand by passing a compatibility check.

You could also switch to scrape data via client side programming, automating a browser to get 1:1 what you see online, too. The difficulty should be obvious frmo this example: Even if you do so, you also onl get one locale shipping cost. The API gives you a concrete paramterization, you don't rely on the html/css not changing, you querey the official data, etc. So overall this has much more benefits than downsides, even in cases an API costs.

Bye, Olaf.

 

[someoneneedshelps]

shipping cost is the only thing left I need, I get everything else, why this is different daily is beyond me... update like the weather man, today its £17.35 its started at £17.33

 

[someoneneedshelps]

Thanks for the info

 

[ChrisHirst]

The simple fact is that the "foreach" must be finding more than one element that matches the criteria.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[someoneneedshelps]

There is only one id for that shipment but commenta off OlafDoschke start another problem, the browser html is not getting exactly what i need according to OlafDoschke, i captcha everything else, een us ifrane to go into the inner page to get the image, shipping is the only problem i can even captcha buy it now

 

[Olaf Doschke]

For example, the asics shoes give me a shipping cost of £11.38 International Priority Shipping to Germany. This will differ for your location, you better use the API simply for the fact this enables you to query the shipping cost for any specific country without the browser making a guess about it via some geolocation machanism via IP or browser request headers, etc. If you do this from your server, it's likely you don't get the shipping cost for the user using your application anyway. And if your PHP runs on a cloud service, you might get different quotes because of making the request from different locations, because your VM(s) can move in many data centers of AWS or Azure or whatever you use.


Chris, the span id really just exists once. Additional to that span there are two javascript calls mentioning the id:

if(powerBid && powerBid > 0  && (powerBid == 2 ||(powerBid > 5 && powerBid < 15)) ){
  new bidSection("bidBtn_btn", "_OPN_POWB_LAYER", "MaxBidId", "[highlight #FCE94F]fshippingCost[/highlight]", "impchSummary", "_OPN_ONLOAD_POWB_LAYER", isRedirect,true,{trackBidBtn:"true", bidTrackingId:"p2047675.l4023"},"false","Enter your max. bid");
}else{
  new bidSection("bidBtn_btn", "_OPN_PB_LAYER", "MaxBidId", "[highlight #FCE94F]fshippingCost[/highlight]", "impchSummary", "_OPN_ONLOAD_PB_LAYER", isRedirect,true, {trackBidBtn:"true", bidTrackingId:"p2047675.l4023"},"false","Enter your max. bid");
}

These two plus the span id are the three only occurances of fshippingCost, though the shipping cost also is in another html section in the "Postage and payments" tab below the bid section.

This javascript is most probably having a runtime effect on this HTML span. I won't deep dive into how the bidSection initialisation influences this, but most probably this updates or creates the span at runtime.

Bye, Olaf.

Edit: I loaded the html via ajax and it starts with the £11.38 cost, before javascript has hands on it, so that JS might only render this HTML or change it's layout. Anyway, the price is locale specific, as you can see from my test.

 

[someoneneedshelps]

I suppose, if im creating this on the fly, i should ask for shipping cost as the only input from the user