Access Internet Data with Delphi 7

[BillKilgore]

Hello Everyone,

I've gotten a request from a person to generate some software that would be used to retrieve historical stock closing prices for several different stocks to be taken from Yahoo's financial section. She's looking for a weeks worth of data at a time for each stock so she can analyse it on the weekend. If I can get the data for her I could probably automate her analyses also as part of the contract. She currently has to go in to Yahoo and obtain the figures 1 stock at a time and has to do it for 27 different stocks each week.

I looked over what I took to be Internet components in Delphi 7, tried some but couldn't figure out how to get them to work.

I would think that a list of symbols fed one at a time into a loop consisting of a variable, start date, and a stop date placed in the Web address should obtain the data (date and value) which would then be entered into a variant array prior to its being written to the appropriate Excel worksheet, would fit the bill.

My problem: Which components to use and what the format of the Web address is supposed to look like.

I've Googled the web looking for info and could find something for whole pages but nothing for specific data value access.

I'd be really grateful for any insights you could provide.

Thank You, Bill K

 

[roo0047]

I think you're on the right track. I did something similar to get information from generated pages. If all the pages are produced by the same form generator, you're in luck. Download a few and examine the html tags. If the data you seek is always between the same tag pairs, it's simply a matter of parsing the html code.

HTH

Roo
Delphi Rules!

 

[YishGene]

I came upon this when looking for a similar webmining solution:

Powtils, a modern Pascal web package

Good luck!

 

[BillKilgore]

Well I've spent 2 days going through sites (Powtils,SourceForge, etc.) trying to find an example of how to format a request for data from a Yahoo page with no luck.
I'm still stuck as to which component to use and how the request inside the loop should look. I did a search of the Tek Tips Delphi site also with no result.
Thanks to Roo and YishGene for your input. Any additional help will be greatly appreciated.

Thanks, Bill K.

 

[roo0047]

Go here

http://www.delphipages.com/threads/thread.cfm?ID=35720&G=35719

Download the zip file, extract, compile, and run the demo. Step through the code. Shows you how to save an HTML doc to file.

You have two chores:

(1) Open the html file you saved (using the demo) with notepad and locate the information you want, say "$300.00" Look at the surrounding tags, something like this:

<stock_closing_price>$300.00</stock_closing_price>

(2) Write your program to do what HttpGet does. Additionally, have it search the saved file for "<stock_closing_price>". Have it save the data that's between the two tags.

HTH

Roo
Delphi Rules!

 

[DjangMan]

A long time ago I used CGI Expert to create a program that read websites. Their main site appears to be down but you can download the components from countless other places.

 

[harebrain]

You're all making it too hard.

(I'm on D7.)

New app. Go to the Indy Clients tab. Grab an IdHTTP component (looks like Earth) and drop it on your form. Go back to the Standard tab and drop a Memo on the form and set ScrollBars property to ssVertical. Drop a button on the form. Put this code in the Button1 OnClick event:

procedure TForm1.Button1Click(Sender: TObject);
begin
  Memo1.Text := IdHTTP1.Get('[URL unfurl="true"]http://finance.yahoo.com/q?s=rimm');[/URL]
end;

Save, run, and click the button. The source of the whole Web page for Research in Motion (stock symbol RIMM) is in Memo1. The price is in a segment that looks like this:

<div id="yfi_investing_head"><div><small>At 1:07PM ET: </small><big><b>123.452</b></big>

That's the hard part: digging through all the hay to find the needle.

Done.

 

[harebrain]

The price is also in here:

<td class="yfnc_tablehead1" width="48%">Last Trade:</td><td class="yfnc_tabledata1"><big><b>122.88</b></big></td></tr>

which is probably easier to find.

 

[lespaul]

I'm with you Harebrain. Here's some code from an application we use that opens the page in a TWebBrowser in a form:

[navy][i]// for automatic syntax highlighting see faq102-6487 
[/i][/navy][b]procedure[/b]  TForm_BrowseDocket.DispDriverHistory(TheWebBrowser: TWebBrowser);
[b]var[/b]
  sTemp: [b]string[/b];
[b]begin[/b]
  Lbl_Msg.Color:= clLime;
  Lbl_Msg.Caption:= [teal]'  Getting Driver History  '[/teal];
  Lbl_Msg.Visible:= True;

  [b]with[/b] Form_BrowseDocketBig [b]do[/b]
  [b]begin[/b]
    Lbl_Msg.Color:= clLime;
    Lbl_Msg.Caption:= [teal]'  Getting Driver History  '[/teal];
    Lbl_Msg.Visible:= True;
  [b]end[/b];

  Application.ProcessMessages;
  TheWebBrowser.Navigate([teal]'about:blank'[/teal]);
  Application.ProcessMessages;
  sTemp:= [teal]'[URL unfurl="true"]http://www/mvdweb/historyByCase?caseno='[/URL][/teal] + strPrefix + [teal]'-'[/teal] +
          strCaseNum + [teal]'&option=h'[/teal];
  [navy][i]{ TODO : Memo1 is just for test: }[/i][/navy]
  Memo1.Clear;
  Memo1.Lines.Add(sTemp);
                                       
  TheWebBrowser.Navigate(sTemp);

  [b]while[/b] TheWebBrowser.ReadyState <> READYSTATE_COMPLETE [b]do[/b]
    Application.ProcessMessages;

  Lbl_Msg.Visible:= False;
  Form_BrowseDocketBig.Lbl_Msg.Visible:= False;

[b]end[/b];

Now I know exactly what to pass to the web browser because we built that application too, but you could use something similar (you'll notice there's some sections that display information in a TMemo too!).



Leslie

In an open world there's no need for windows and gates

http://www.ubuntu.com

 

[harebrain]

Leslie,

We're thinking more alike than you guessed: I put a WebBrowser component on the form in my original example and in the OnClick handler did a Navigate to the same URL. Side-by-side, I had the HTML source and rendered page. I omitted that here for the sake of simplicity and emphasizing the crux of the issue.

 

[harebrain]

And it looks like this:

http://finance.yahoo.com/q/hp?s=RIMM&a=02&b=30&c=2008&d=03&e=3&f=2008&g=d

would be an example URL for a week's worth of closing prices. I wonder what genius decided to number the months from 0 to 11!

 

[whosrdaddy]

Has anyone ever considered connecting to a webservice for this.

much easier

one example:

http://www.gama-system.com/webservices/stockquotes.asmx?op=GetLatestStockDailyValue

all you you need is to call the webservice and you get a nice formatted XML response.

/Daddy

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[roo0047]

Bill - My method was brute force for lack of answers to your despiration.

All - What took you guys so long!?!

BTW: HttpGet is still worth a look to see how it works for learning aspect and a very small footprint. I used before any of these other (better) options were available with D4.
I was also on dialup (4000+ pages/sites) so downloading over-night for processing during the day was more economical.

<excuse>END</excuse>

 

[harebrain]

So, Daddy, take my example and replace the line in the OnClick handler with (this is supposed to be one line: string literal)

Memo1.Text := IdHTTP1.Get(
'[URL unfurl="true"]http://www.gama-system.com/webservices/stockquotes.asmx/GetLatestStockDailyValue?strStockExchange=NASDAQ&strStock=RIMM');[/URL]

The result is a nice, uncluttered response. There are other functions there that would supply prices over a time interval as OP needs.

 

[BillKilgore]

Hello Everyone,

Please excuse the somewhat belated response. Here’s why. We landed in Honolulu on Apr. 3rd after coming east over night to find that the airline providing my connection to the mainland, ATA, had gone out of business!
But,since I was in Hawaii I decided to recover from the wrenching experience and the additional expense by taking an unscheduled vacation and forget about computer-related things for a couple of weeks.

During that time I purchased a ticket from Delta where those friendly folks, sensing opportunity, were gracious enough to charge $1300 to fly me to Atlanta. The $1300 was in addition to the $800 I had paid ATA for the same service. The lesson her: Carefully study the financial statements of any airline you plan to purchase tickets from.

Speaking of finances, I’d like to thank all of you who responded to my questions. I knew there had to be a fairly straight-forward way to access those data. The IdHTTP component was the key.

Harebrain, I took the procedure you included, plunked it in wholesale and voila, it worked! There appears to be a lot of parsing to do to find the “needle in the haystack” though that shouldn’t be that tough.

While typing this I discovered another possible approach that seems to work pretty well.

I just copied the downloaded data from the Memo1.Text and pasted it into a blank Excel worksheet. Saving the Excel file as a CSV file reduces everything to so many strings making the data really easy to locate. Not the most elegant but it should work and it would reduce a great deal of the parsing.
I haven’t tried it with more than one page of Yahoo’s historic closes so I don’t know yet how it will do with multiple pages. We’ll see.

Anyway, I think I’ve got what I was looking for so once again, thanks for all the help and insight.

When I get it into a presentable form I append it to this thread.

Bill K.

 

[harebrain]

Bill, I'd still look at

http://www.gama-system.com/webservices/stockquotes.asmx

Its output is *much* easier to handle than the Yahoo page, unless you're stuck with Yahoo for some reason.

(ATA, ouch!)