Choose database for large amount of data acquisition

[Wavelet]

I need to handle large amount of experiment data output from a 8-channel instrument (approximately 400Mb per day and it works continuously all the time). And it is required the data be real-time processed every 2 or 5 minutes while acquisition is going on. Someone told me that I should use database to manage the data acquisition and processing. I am fairly familiar with C++ Builder but poor in my database knowledge. My questions are1) What kind of database is suitable for the application? (2) Which one should I use to create and handle the database: TTable or TQuery? Thanks.

 

[2ffat]

It may depend on which Builder you use, Pro ro Enterprise. With Pro you are going to be limited to using BDE, OLAP, etc. With Enterprise you can use any of the "professional" data bases like Oracle, MS SQL, MySQL, Interbase, etc.

It sound like you will want something with a small footprint, that it, something that does not take a lot of overhead. That eliminates large DB like Oracle and MS SQL. You probably want something that is easy to install and use. As you look over this forum you can see that BDE has some drawbacks with deployment. In addition, Borland has said that BDE is not going to be supported in future releases.

That means you need something that will be easy to administer, small overhead, supported, and (probably) inexpensive. You have several option left:
[tab]Interbase: Comes with Builder but I've found it a pain to deploy. You may have to buy a license in some cases.

[tab]Firebird: A "free" version of Interbase.

http://firebird.sourceforge.net/

[tab]MySQL: Cheap, relatively easy to install. If you have "Pro" version of Builder you have to use something like OLAP or a third party component to talk to it.

http://www.mysql.com

[tab]Advantage: Very good. Supplies the components with the DB. You do have to pay for this but it is not outrageous. Easily upgradable.

http://www.extendedsystems.com/ADS/default.htm

[tab]Flashfiler: My new favorite. I plan to use this for all my programs that used BDE. This is now a freeware so support may be intermitant. Very easy to set up and deploy. Easy to multi-thread. Comes with it own components or you can use Borlands.

http://sourceforge.net/

Search for "flashfiler" or "turbopower."

All the above should be able to handle the number of entries you indicate. Anybody with any others, more comments?

As far as whether to use TTable or TQuery, it may depend on which DB you use. Some DB cannot use the TQuery to update the DB, you have to use their supplied components or TTable. Some of the supplied components are easier to use than Borland's so look into that, too. James P. Cottingham

When a man sits with a pretty girl for an hour, it seems like a minute. But let him sit on a hot stove for a minute and it's longer than any hour. That's relativity.
[tab][tab]Albert Einstein explaining his Theory of Relativity to a group of journalists.

 

[Wavelet]

Hi 2ffat, thank you for your help. It seems that Adaventage or Flashfiler may be my choice. Now I need some time to get familiar with them.

 

[butthead]

Last months issue of C++ Users Journal had a topic about a lossless compression algorithm that may be useful to you. Or was it the month before???

go to

http://www.cuj.com

tomcruz.net

 

[LastCyborg]

To answer the second question, let me tell you that depends of what kind of data manipulation will you do.

If you have a BD with 1,000,000 of records, and use a TTable, the application will run very slow, besides there are data that is not necessary to be there. Tha advantage is that the engine of any BD will point to the data wich you are looking for very fast.

If you choose to use a TQuery, you will write less code than with a TTable, but you'll pay with velocity, because your aplication must exceute the SQL statements.

I recommend the TQuery, because is easier to write SQL code than make you own function to get what you want.

---LastCyborg---