VFP read sequential files 1

[Jan Flikweert]

Hi all,

I hope I spelled the word sequential files correct.

I attached an example of this kind of files. How to read this in VFP? Line by line filetostr seems not to be the option. wscript.shell stdin read is also not clear and not well documented.

This is a file from the open source program OSCAR. And it contains data from my personal CPAP device regarding sleep apnea.

Kind regards,

Jan Flikweert

 

[Tore Bleken]

Depends on what you want to do with the data. To read the data, you may use VFP's low-level file functions.

 

[Jan Flikweert]

Tore Bleken (Programmer)
7 Dec 23 08:32
Depends on what you want to do with the data. To read the data, you may use VFP's low-level file functions.

Tore,
Thanks for your reply.
I suppose text like this readed with notepad: “3 @á3 À–5 @Õ5 °ì5  B6 àa6 ˜6 °6 €Ö:  $; P[; p©; à; 6< Ú< @(= ðÉ@ ?A VA p…A àœA ÐòA @
B °!B 9B `XB ÐoB à–B püB ÐWF €ŽF ð¥F 0ÅF  ÜF €G ÐQG €ˆG ˜G ÐÎG pÞG P
will not be readable with VFP low-level funcions.

Kind regards,

Jan Flikweert

 

[Mike Lewis]

Yes, it will be readable with low-level file functions. But that's not the point. As Tore said, the question is what you want to do with the data. Or, more importantly, how you interpret the data.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Lewis]

I don't know anything about OSCAR, but a quick glance at their documentation finds this:

You can export all your notes as an XML file for backup purposes or additional analysis. Use the menu option File | Backup Journal.

I don't know if this refers to the file that you are asking about. But if it does, you shoud export it to XML, which is more easily readable by VFP. To do so, use either XMLTOCURSOR() or treat the file as a text file and parse the XML in your code.

If this is not relevant to your file, read the docs for yourself to find out about other export formats.

UPDATE: And here's a link to a discussion where somone else seems to have a similar question:

https://www.apneaboard.com/forums/Thread-OSCAR-CSV-Export-Wizard

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

Btw, you can read any file with FILOETOSTR() unless it's too large. But you expect to find something readable in it, it's likely a binary file in which even numeric data isn't stored with digits 1-9. Just like integer fields in VFP DBFs are also not storing 9 character numbers but 4 bytes binary value.

So you'd have to know the binary structure of the file to make something out of the bytes you get from it. Low level file functions just make it easy to read in byte by byte or any other length chunks you want, instead of reading the file in one "gulp". But you get the same bytes from fread or fgets as you get from filetostr, that won't also change whatever other trechnique you use. The bytes of the file are the bytes they are, it's the interpretation you won't always find documented.

And even if you find which byte numbers of files contain what information exactly, it seeems to me the alternative output of XML is much easier to process, not only with VFP, but in general.

So the question just becomes where the export feature of OSCAR is. Aside from that feature, if OSCAR is this application, it already is for graphing the sleep data:

https://www.sleepfiles.com/OSCAR/

So isn't it giving you the desired results, can't you zoom into details or what is wrong about OSCAR? Are you perhaps not having compatible CPAP data?

Chriss

 

[Jan Flikweert]

Hi all,

It is difficult to explain. Of course I saw the export .csv/xml function. .csv does not export all calculated fields,no detailed data from all sessions at any time. .XML only my name. And indeed OSCAR produces much graphs/info. The next image illustrates things can always be done better/cleared/detailed. And if not detailed enough, name it. The image shows sleep analyze from one night, produced by Samsung Galaxy 4 smart watch.

My primary goal is to get more/clearer analyses of my sleep apnea CPAP therapy. This goal can also be achieved by reading the .edf+D files from the SD card from my device.
My second goal is to let other people on the forum enjoy this.

Using csv/xml for other people is not user friendly and requires a user action depending on the software.
The graphs/info in OSCAR is great, but reading the forum people sent there output with questions about is. The anser I saw was other software is needed, they cannot see it in detail.

Using the binary(My subject description is wrong, it are binary files.)files depends on files wich are default present on the computer.
I understand it is a matter of how much bytes per value. The pro is there are for day two files: events/summaries.

Kind regards,
Jan Flikweert

 

[Mike Lewis]

Thanks for that extra information, Jan. But I'm finding it hard to understand.

Your wrote: ".csv does not export all calculated fields". Well, that's almost certainly by design. In general, an application does not need to export calculated fields because thye can be calculated by the software that imports them. That's the definition of a calculated field.

You mention ".edf+D files". What exactly are these? You don't mention them in your original question.

You say: "Using the binary ... files depends on files which are default present on the computer." It's notthe default for DOCX files)? If so, there is no reason for that to be a problem. The ability to process any give file type does not depend on the presence of a default application.

EDIT: You say: "Using the binary ... files depends on files which are default present on the computer." What do you mean by that? Do you mean the default application for a given file type (in the same way that Word is the default for DOCX files)? If so, there is no reason for that to be a problem. The ability to process any give file type does not depend on the presence of a default application.

The point is that, if you want to process this data in a VFP app, you will need to know how to interpret the data in the export file - whether that is CSV, XML or the binary file that you posted in your question.

If the CSV file is missing calculated fields, then you will need to know how to calculate the values in those values from the other fields in the file. That's a perfectly normal thing to do.

If the XML file is only showing your name, I suspect there is something wrong in the method you used to export it from OSCAR. After all, why would OSCAR give you the ability to export its data and then restrict that data to a single field?

If you want to use the binary file, you will need detailed documentation on its internal format. You won't find that here. If that information is published anywhere, it is most likely to be in the OSCAR docs.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

You mention the files are EDF, well that's an important detail, isn't it?

Well, The link I gave points out OSCAR is "exploring data produced by CPAP and related machine" and the description text says "or the viewing of the high-resolution sleep data that is generated by the CPAP and stored on the SD card"

So there are files on the SD card, and by what you say they are EDF format files. Searching for CPCP files I came across

https://www.apneaboard.com/forums/Thread-file-formats

which also points out EDF and links wikipeadia

https://en.wikipedia.org/wiki/European_Data_Format

Which in turn points to

https://www.edfplus.info/

with specs (specifications) in

https://www.edfplus.info/specs/index.html.

So now you have some pointers about what this is, not only sleep data, but a binary format for time series data of many different things, not only sleep data. It's not just, as you expect a series of numbers with a fixed byte length. I mean how could it, there would at least be a prespan about what time the data series starts and likely also what data that is, renge of values, perhaps, and more info.

Many chunks of the file will perhaps be 4 byte binary values, that could be the case, it could also be floating point numbers with a legth of 8 bytes or more. VFP has some functions for converting binary to values like CTOBIN(), which has some options to coinvert words (2 byte length) or normal integers 4 bytes, signed or unsigned, but before you dive into all this:

You also see the EDF Downloads section

https://www.edfplus.info/downloads/index.html

shows there is more software than just OSCAR to deal with EDF data and before you reinvent the wheel I'd look into that software list and what it can do for you or others in the forum you want to work for.

There's no simple decoding of binary, binary is just the bits, you have to know what to expect where in a file to interpret a binary file correctly and there's no decoding information included that a general filestream function would use to produce "human readable" or otherwise simpler to understand data. You expect too much automatisms about this to exist just looking for another way to reda in the files. As said, the bytes are the bytes, no matter how you read them.

I think you could add yourself into the list and provide yet another software solution, okay, but this could carry you away into many different details aside frm the main data series in these files and the different formats EDF and EDF+ and whatever other formats other watches or devices produce.

With the limited concepts you have about reading files, it's a steep learning curve ahead, going into very detailed things as how data types are stored. Even a simple type as an integer can be stored with lowest significant byte first or last, aka as endianness.

You'll put yourself into elementary computer science classes, not to discourage you, but it can get very fiddly and then without guarntuee to be able to generally decode any EDF or EDF+ files.



Chriss

 

[Chris Miller]

All that said, I'm pretty stumped you only get your name exported into an XML file, that doesn't sound right to me, there should be more export possibilities. Well, and that also only concerns OSCAR, the EDF site3 lists much more software for EDF file processing and this should be consulted first to find something not being the end processor of generating graphs, but just turning the binary format into series of data easier to process. So look for preprocessing software that at least already solves the task of reading and "understanding" the EDF files and what they contain in detail.

Chriss

 

[GriffMG]

Looking at the spec for the EDF, and comparing with the file you supplied... I don't see any correlation B-(

Is it encrypted or somesuch?

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

If this is specifically the start of the file, I agree with you Griff, because by the EDF specs the first 8 bytes should give version information, then 80 bytes patiend identification etc., the specs don't even tell details about what format these 80 bytes have, whetehr they are just a name or some other code.

Anyway, when it comes to the DATA RECORD of the specs it will just be a series of integers. And no byte is disallowed in an integer, so those bytes are whatever the data needs them to be. This "text" could be from the middle of the file, and this "garbage" then is normal. It's not failure to read the bytes, an editor like notepad just does not interpret the bytes, it just shows each byte as one character, and an ANSI codepage has 256 characters, many of them are letters, but most of them are not, some of them are not even printable characters but control characters. The data just isn't text, that's all you learn from a text editor.

Chriss

 

[GriffMG]

From the examples in the specification I was examining, the format should be human readable, I took the integers to be as they are in a .dbf file...

The examples included signed non integers



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Well, Griff, if you store integers ni a DBF you get the same garbage. Do you expect to see digits 0-9? Or are you confusing a browse window with a low level file access? In cells of a browse window the binary stored integers are turned into human readable numbers in digits.

There is anumber format in VFP that is humany readable even if you open a DBF in notepad, you'll see the numbers, that's the numeric format N(x) using x bytes and storing the number as one digit per byte. But integers are 4 bytes.

Here's a small demo:

Local lnHeader, lnRecsize, lnFirstRecordOffset

Create Table ints (number integer)
lnHeader = Header()
lnRecsize = Recsize()

Insert into ints values (123509032)
Insert into ints values (235423909)
Insert into ints values (154552397)
Use

lcDBF = FileToStr("ints.dbf")
lnFirstRecordOffset = lnHeader+2 && actually +1, but the first byte is the DELETED() mark, which we don't want to read

For lnI = 0 to 2
   lcBinaryInt = Substr(lcDBF,lnFirstRecordOffset+lnI*lnRecsize,4)
   ? lcBinaryInt, CToBin(lcBinaryInt,"4RS")
EndIf

This is "decrypting" a dbf without using VFP workarea, just FILETOSTR(). I'd not call this deccrypting, actually, as it's not really encrypted data, it's just stored in binary format, 4 bytes make up one integer value. This only needs to use the rough knowledge about the dbf structure, that a DBF has header with the length HEADER() - actually a VFP function you could read about in the help - and records have RECSIZE() length including the 1 byte necessary for the deletion mark of records. It would also include the bytes necessary to store which fields of a record are NULL, if nullable fields would exist, but that's just details.

All in all reading 4 bytes from lnFirstRecordOffset+lnI*lnRecsize gives you the 4 bytes that represent the integer number and outputting that as text looks like "garbage". It's not an encoding or encryption, it's just the characters those 4 bytes have. But they are meaningless in terms of what the actual number is they represent, for that you need a conversion function like CTOBIN, that's all there is to it. The simple truth is they are just 4 bytes or 4x8=32 bits. In bits it's easier to output them as a series of 1s and 0s, bytes would also usually better be represented as 2 hexadecimal digits 0-f just as hex editors do, which will just look less like garbage but still don't give you the decimal mrepresentation you're so used to.

What bytes mean in a binary file is not deducable from the byte values alone, you have to know how a file is composed and where to find what. As I know here in case of a DBF file where to read the bytes that represent the four numbers I inserted into an integer field.

Chriss

 

[GriffMG]

It's been a while since I looked to be honest, maybe I am thinking of a very aged format...

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[GriffMG]

[URL unfurl="true"]https://res.cloudinary.com/engineering-com/image/upload/v1701964401/tips/Noname_i48h2v.bmp[/url]

Ermmm, Chriss

I just made a free table and...

It is stored in human readable form when you open it in a hex editor or even notepad.



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Griff, data type N is not I.

Chriss

 

[GriffMG]

True

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Jan Flikweert,

to get back on track of the EDF file processing, you can take from my DBF interpretation example, that the function CToBin(lcBinaryInt,"4RS") can be used to convert 4 bytes to an integer value you can then coninue to work with. The problem I see is that the specification isn't very detailed even though

http://www.edfplus.info

seems to be the defining homepage with the information about this file format, so it's quite cumbersome to understand what data series you have in an EDF file and from which offset to read the actual data.

Well, as said there is a list of downloads of software that deals with these specifics and more. I'm sure you'll find something that can turn an EDF file into an easier to process format, may it be something as simple as csv.

The most important part of the header seems to me these 3 lines of the specs:

8 ascii : number of data records (-1 if unknown, obey item 10 of the additional EDF+ specs)
8 ascii : duration of a data record, in seconds
4 ascii : number of signals (ns) in data record

They specify the length of the following data record, the number of records and the number of signals within each record. One detail problem is that the header has a variable length with a section per "ns", per number of signals each record of the data record has, so you can't just start off at byte X and know the rest of the file is just a series of integers. And even if you get there, as far as I understand the header data the literal integer values are not the actual measurements, they have to be scaled to min/max values and their units defined in the header section before a data section, and last not least after a record section an EDF file could continue with a next header followed by another data series.

EDF+ format complicates this further. If you want to go from square 1, okay, but it will be cumbersome until you'll be sure from which byte offset to read which number of bytes with which meaning from an EDF file.

Chriss.

 

[Jan Flikweert]

Hi all,

The .001 and .000 are no EDF+ format.
The pro of these files is : they are always present.

I have found a command line program which converts edf to csv. Not the most elegant way, but so be it.

The con of edf is that my brand CPAP Medres is the only device with this format. For me this will do. Other brands are not readable.

Thanks for your efforts!

Kind regards,

Jan Flikweert