cgi script to send data via POST needs a tweak !

[StroudPete]

Background first. I'm a (forms) vb.Net developer but come from a Unix background. I have a working knowledge of Perl, good enough to write a quite logically complex cgi script to do searches on flat file data for my website. I do not have a great deal of practical perl knowledge - just what I taught myself to build a few scripts (the search is several hundered lines). I can grasp the language constructs and code, but I don't properly understand the inter-relationships when moving between browser pages / scripts and browsers (again, enough to "get" info from my webpages and process, and to pass to new scripts - oh, and write html code, obviously).

I want to intiate a search on my phpbb board. Using the code from the built-in search page I have constructed a web page that simulates the phpbb search page, whilst looking like the rest of my site, and being understandable to non techie users. The data is sent via "POST". Because I' want to fiddle with the search terms the user enters, I'd like to send the data to a cgi script, fiddle with it and then initiate the php search.

The problem. I can initiate the search using the standard "GET" method of sending data, and the results are almost perfect. The only thing wrong is that I get the results listed by post rather than topic. When I use a "POST" method, the results are correct but not formatted with colours and there are no images (IE6) or the browser just lists the html code (Firefox 1.5). I haven't tried sending POST data before, and only scabbled together my script after a lot of googling - I don't really understand the concept enough to debug it.

The help I'm looking for! Since this isn't a phpbb board, I don't expect any help on making the GET script work perfectly, and since the phpbb html code uses POST, perhaps it can't be made to work 100%. Hence I'd like to know what I'm missing in the POST script - obviously I'm missing some vital code!

Here's the snippet of my html page which gives perfect results:

      <form action="[URL unfurl="true"]http://ccgi.quaffersoffers.co.uk/QOforum/search.php?mode=results"[/URL] method="POST">
        Find tasting notes containing
		<input class="INPUT_TEXT" type="text" name="search_keywords" size="25">
		<input class="BUTTON" type="submit" value="Search" onmouseover="this.className = 'BUTTON_MOUSEOVER'" onmouseout="this.className = 'BUTTON'">		
		<br />	
		with:&nbsp;	
		<input class="RADIO_BUTTON" type="radio" name="search_terms" value="all">
		<b>all</b> of the words
		<input class="RADIO_BUTTON" type="radio" name="search_terms" value="any" checked="checked">
		<b>at least one</b> word
		<!-- hidden FIXED VALUES -->
		
		<!--   search ALL forums in all boards (we only have one board) -->
		<input type="hidden" name="search_forum" value="-1">
		<input type="hidden" name="search_cat" value="-1">
				
		<!--   search topic title and message  -->
		<input type="hidden" name="search_fields" value="all">
				
		<!-- sort by date & time descending -->
		<input type="hidden" name="sort_by" value="0">
		<input type="hidden" name="sort_dir" value="DESC">
		
		<!-- results as TOPIC not post -->
		<input type="hidden" name="show_results" value="topics">

		<!-- return 200 characters -->
		<input type="hidden" name="return_chars" value="200">
		
      </form>

Here's my test cgi GET script (works except the show_results=topics parameter seems not to be processed) :

#!/usr/bin/perl --
use CGI;
use CGI qw /:standard/;
my $query = new CGI;
my $nextUrl = "[URL unfurl="true"]http://ccgi.quaffersoffers.co.uk/QOforum/search.php?";[/URL]
my $theParams = 'mode=results&search_keywords=chardonnay&show_results=topics&search_terms=any&search_forum=-1&search_cat=-1&search_fields=all&sort_by=0&sort_dir=DESC&return_chars=200';
print $query->redirect(-url => $nextUrl . "?" . $theParams);

Here's my test cgi PUT script (works on all parameters, but IE6 shows page unformatted and no images (no "home"/"root"?) and Firefox just displays the html code):

#!/usr/bin/perl --
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = LWP::UserAgent->new;
my $url = '[URL unfurl="true"]http://ccgi.quaffersoffers.co.uk/QOforum/search.php?mode=results';[/URL]
my $form = {
        search_keywords => 'chardonnay',
        search_terms => 'any',
        search_forum => '-1',
        search_cat => '-1',
        search_fields => 'all',
        sort_by => '0',
        sort_dir => 'DESC',
        show_results => 'topics'
};
my $response = $ua->request(POST $url, $form);
print "Content-type: text/plain\n\n";
print $response->is_success ? $response->content :
$response->error_as_HTML;

Thanks for reading all this!

Pete Jones

 

[KevinADC]

print "Content-type: text/[b]plain[/b]\n\n";

try:

print "Content-type: text/[b]html[/b]\n\n";

- Kevin, perl coder unexceptional!

 

[StroudPete]

Thanks for replying Kevin. In no way knocking your help, it's obvious once it's pointed out, like so much in IT! The effect was to render the page in the same way in Firefox as in IE6, so that's a browser implementation inconsistency out of the way.

I'm still left (IE & Ffox) with the page being unformatted and no images. So now it's a case of "what piece of information is not being passed by my script that is being passed by the html code". I'm assuming it's something that browsers do without visible instruction that needs to be emulated by my script. I'd google for it if only I knew what it was I don't know!

 

[KevinADC]

OK, what you can try is this, don't print any http header at all in your cgi script:

my $response = $ua->request(POST $url, $form);
print $response->is_success ? $response->content :
$response->error_as_HTML;

- Kevin, perl coder unexceptional!

 

[StroudPete]

Thanks again Kevin. Unfortunately that change meant the script returned nothing at all - it just timed out! However, I've got my teeth back into this.

I compared the source of the page returned by using the html query and that returned by the cgi query. Eliminating the "sid" string, the source was identical.

Next I used a Firefox plug-in to alter the source of the page in a browser sidebar. All the links are, of course, relative ones. So (using the cgi imitiated results page) I amended one image link and added in the full path name. The image appears. Next I added in a css definition with a full path (this editor doesn't show the very top code, so I couldn't amend the original line). Bingo, the formatting appears.

So, what is missing is the browser knowing what page everything is relative to. I don't know where that comes from, as the html code that works is initiated from my web site (

http://www.quaffersoffers.co.uk)

and the scripts are on the cgi server (ccgi.quaffersoffers.co.uk) - which are separate servers. This indicates to me that a hunch that the "referrer" might be important wouldn't be correct, as they are different ...... unless something in the forum code that produces the results page ... oh, I don't know. I'll carry on poking and googling. If I solve it, I'll post the solution.

 

[KevinADC]

whe I tries it sans the http header being prined it returned the requestd page fine, but of course sans the images and cs6s defintions for the reason you are aware: relative urls.

I noticed the html code was a mess. XHTML 1.0 mixed in with html 3 ad html 4 tags/attributes with an html 4.01 header.

- Kevin, perl coder unexceptional!

 

[StroudPete]

The html code for the results team is produced by phpBB, though I've "tidied up" (spaced out and indented) some of the code in the phpBB templates (and hence the results page) so I could see what was going on in the tables!

On the main matter of getting the rel.page to be recognised, I tried all sorts of things with referer and other variables, to no avail.

Eventually, I gave up. I've achieved what I wanted to do (which was recognise a checked radio button and add a string to the user-input text string) in the original html page using javascript. I was trying to avoid java as my site is almost completely java free, and I don't know java! However a while with google and a java book I have for emergencies got me there.

It was so much easier than using perl, even if you know what you're doing (add an onSubmit clause, tiny script with one if statement!!).

Kevin, if you read this, thanks for you efforts. I did learn some useful stuff along the way!


Pete