http://www.webthangs.com

have not done many PHP projects but I always think that PHP is really good for web programming even though I mostly program in Java and .NET.

I am not a good PHP programmer, but to write a simple web page scraper in PHP is very straightforward.

First, I defined my HTTP get/post function

function http_post_content($url,$data) {
$data = http_build_query($data);
$aContext = array(
‘http’ => array(
‘proxy’ => ‘proxyserver:8080′,
‘request_fulluri’ => True,
‘method’=>’POST’,
‘header’=>
“Content-type: application/x-www-form-urlencoded\r\n”
.”Content-Length: ” . strlen($data) . “\r\n”,
‘content’ => $data
),
);
$cxContext = stream_context_create($aContext);
$content = file_get_contents($url, FILE_TEXT, $cxContext);
return $content;

/*
$fp = @fopen($url, ‘rb’, false, $cxContext);
if (!$fp) {
throw new Exception(”Problem with $url, $php_errormsg”);
}
$content = @stream_get_contents($fp);
if ($content === false) {
throw new Exception(”Problem reading data
from $url, $php_errormsg”);
}
*/
}
In the http array I can define the proxy server settings, method which can be post or get, and the content.

I use file_get_contents to retrieve the web page instead of cURL, which is more powerful.

To test it,

$url=”http://biz.thestar.com.my/marketwatch/main.asp”;
$data=array(’bns’=>’2′,
‘clp’=>’1′,
‘klseViewDate’=>’1/30/2008′
);
$content = http_post_content($url,$data);
$htmlDoc = new DomDocument();
@$htmlDoc->loadHTML($content);
$xPath = new DOMXPath($htmlDoc);
$counters = $xPath->evaluate(’//table[@id="Table2"]
/tr/td[2]/center/table/tr/td/span
[@class="text"]/table/tr/td/a’);
for ($i = 0; $i length; $i++){
print($counters->item($i)->nodeValue.’‘);
}
I pass in the URL and the post data, then I use XQuery to retrieve the information that I want.

Note the HTML may not be well-formed, so I suppress the warning by prefixing a @ in front.

One of the catch here is the correct XQuery/XPath to be used. You can always find out using Solvent, which is a very good FireFox plug-in for web page scraping.
quote]

Wayne

I don’t have any idea of how to use this script. Can you please give us some detailed instruction for doing it? I have messaged you on Digital point forum about this. Please guide me.

It’s actually really easy, just copy the code and save it on a file (youtube.php for example). Then wherever you want to insert a Youtube video related to a keyword, just add this:

require_once(’youtube.php’); // assuming your file is called youtube.php

And then just wherever you want to see the video add this:

youtube(’bmw’); // this will include a youtube video about BWM

I’m getting this error.