String Space questions

[tedsmith]

I am considering an app that has to match up continuous GPS data being received as a stream from up to 5000 buses on a 1gb fibre optic network.
This data is collated by others and I would received it at the rate of about 20 buses per second.
As each packet is received I want to look up a table to establish if the vehicle is where it is supposed to be from a schedule and modify as database record if it is not. This requires a computation 20 times a second.

The Schedule database can have 100,000 records and consists of way points where each bus should be every 2 minutes throughout the day (the bus timetable)

I can easily do this using the usual dbOpenRecordset stuff but it is much faster just using simple string arrays and is all done in memory. Results only need to be saved every few minutes so the disk is not being thrashed.
Problem is MyString(100000,5000) is far too big!

Any ideas as to how I can do it in memory?

Is there a way of measuring how much string space is remaining in a given vb6 app? (not just available RAM)

 

[dilettante]

Even if your String array had only 1 character in each String element you'd need 10 * 100000 * 5000 bytes of memory.

2 bytes for the pointer to each string, 4 bytes for the length of each string, 2 bytes of NUL termination, and 2 bytes for the single character in each String.

From here that looks like 5,000,000,000 bytes.


If you have 5,000 buses and 20 waypoints that would give your 100,000 "records" so are you sure you need 100,000 x 5,000?


In any case I believe a database-based solution will serve you better. Your mention of what looks like a DAO method suggests you haven't looked at using them in a very sophisticated way.

While DAO can have some benefits when using a Jet MDB, getting at some of the "tweaks" you need for many performance profiles isn't as easy as it is using ADO and the Jet 4.0 OLEDB Provider. And for the most part they aren't as well documented. For example you might increase the DBPROP_JETOLEDB_PAGETIMEOUT value from the default, set DBPROP_JETOLEDB_RECYCLELONGVALUEPAGES to True, etc. to reduce unnecessary disk accesses.

If you are going to use a Jet database for this ideally you'll want to use a single connection, opened exclusively. This eliminates all locking overhead. While this means another program can't query it while opened this way, they also won't see your "Strings database" or get consistent results reading your flushfile on disk.

Async writes can also help avoid tying up your program's UI thread, which is where all of your code you write in VB6 must run.

If you need to allow secondary programs to query the data while your program is in use, open it denying shared writes in your updater and open it read-only in query programs.

 

[CP60]

>For example you might increase the DBPROP_JETOLEDB_PAGETIMEOUT value from the default, set DBPROP_JETOLEDB_RECYCLELONGVALUEPAGES to True, etc. to reduce unnecessary disk accesses.

You can set these under DAO as well.

 

[dilettante]

Of course you can, but people seldom do. My only point is that by using a database carefully no matter what database you use you can tune performance to your problem.

In this case the original problem might be a better fit for something aside from Jet that conveniently handles very large databases. This might be a client/server database like SQL Server, MySQL, etc. or it might be another embedded database such as SQLite.

 

[CP60]

True. (as mentioned: depending on size, location and # of concurrent users)

 

[tedsmith]

Thanks
I actually have an old working database version that I cheat with a little. It is in use on a much smaller system but I have been asked to look at extending it to cover the whole city bus network as their current system designed by experts keeps telling lies!

What I have done so far is to have a smaller 'working database' that contains only the next hours worth of waypoint times so it is about 1/10 the size of the master timetable.
Buses are never more than 15 minutes late or early so this covers all the buses actually on the road at any time and no route is longer than 45 minutes.
I update this smaller database every 5 minutes and flush it of buses that have reached their terminus with two simple execute statements.

The 100000 record database contains each route, the time the first departure in each route starts, the time it is expected at each waypoint and a blank column added for bus ID numbers 1000 to 9999.
As each bus is sensed,(20 per second) this smaller database is searched for a the closest GPS location and the waypoint found. The bus ID is written to the database column and all other instances of that particular run of the route so all subsequent sightings only have to search for the that bus ID.
This speeds up searches considerably except that I have to read and write to the same record in the same step.

The difference between the scheduled waypoint time and sensed time tell if the bus is running early or late at any time of the day and used to update hundreds of roadside passenger signs and shortly a SMS service that passengers can phone.

That's why I thought strings might be faster?

Regarding measuring string space, are you suggesting that the number and size of strings in vb6 is only limited by the size of my RAM? IO though there was an upper limit regardless of the RAM.
Also what happens when you get to over 32 bit addressing with vb6 in a 64bit OS or is this irrelevant?

 

[gmmastros]

I'm just curious... what city is this for? 5,000 buses seems like an awfully large number.

Coincidentally, I will begin building a similar system where I will be getting GPS data from buses and matching the time and position with expected times and positions so that I can raise alerts when a vehicle is early, late, or deviates from it's expected route.

Even though my beta site only has 120 buses, I will be receiving positional updates every 10 seconds so I will likely get 12 updates per second, which is similar to your 20 per second.

-George
Microsoft SQL Server MVP
My Blogs
SQLCop
twitter
"The great things about standards is that there are so many to choose from." - Fortune Cookie Wisdom

 

[dilettante]

If you use an array of String type, each String value is limited to about 2GB.

http://msdn.microsoft.com/en-us/library/aa263420(VS.60).aspx

VB6 programs are 32-bit so they run under WOW64 on a 64-bit Windows OS. With a hack they can have about a 3GB address space but normally they only get a 2GB maximum.

Even if you could feed them a ton of RAM and they could take it, a lot of your data will get paged out to disk. Windows is not a single tasking real-time OS and isn't tuned for that. This is one reason you want to dedicate a whole box to SQL Server, it uses tweaks to change Windows' behavior in a way that other applications get starved for resources. Those tweaks are not readily accessible to us for our programs though. SQL Server Express and Developer are somewhat "detuned" to limit this.


Wild guess on the 5,000 buses number:

Is this really a design-limit target instead of an actual figure? Something to aim for so you don't risk obsolescence?

 

[tedsmith]

Brisbane Australia. We already have over 2000 government owned buses growing by about 500 per year and about 500 privately owned buses. I think 5000 will be bus saturation point but who knows?

Having a population over 2 million spread over nearly 6000 sq km., the population density is quite low compared with most similar overseas cities.

So we couldn't afford an underground train system, instead we have a network of dedicated elevated busways with "bus stations" every 2 minutes traveling time that functions just as quick as a train. As well, many buses only go so far and break off to distribute passengers instead of having to catch cross connecting buses.

The system originally used 400khz radio transponders picked up by 500 traffic light loops to tell if a bus is running to time but they are converting this to GPS that each bus already has for its smartcard ticketting system.

A few years ago I wrote the program for the passenger info LCD screens at the CBD stations and 10 other major "station" interchanges (total of 40 bus stops so far). It interfaces with the major timetable database and the transponder system to show real time arrivals. It also allows multimedia mixed with bus info like an airport check-in.

The have called me back from retirement to look at interfacing my old passenger info programs with GPS.

That is the problem!

I would be interested in sharing any info with anybody with a similar project.

There seem to be many ways of handing this however one big problem is anticipating when a bus will arrive at any given waypoint is that in any given route a bus can be late for some way points but early for others depending on rapidly changing traffic conditions. This is necessary for each individual bus stations passenger info screens.

My bus info display and control systems have been running since 1987 when I built the first one using DOS basic and a home made "touch screen" using a grid of infra red LEDs.