Binary raw data file conversion

[jpnovak]

I just completed the first version of a micro-SD card datalogger for MSII. My output file is currently a CSV of the 113 byte string output. I saw that TunerStudio is capable of the file conversion process. Can you provide some file format guidelines to import the data from binary numbers?

Thanks.

[LT401Vette]

http://www.efianalytics.com/TunerStudio ... talog.html

You can generate a sample file by doing a data log from TunerStudio. Just change the file name extension to .frd and TS will log in frd format.

You can then convert those to a standard msl file from the menu:
Data Logging -- Import / Conversion

[jpnovak]

Phil,

I tried to setup a format. Can you make a few clarifications for me?

The string output is in hex? I am currently logging in binary. I can switch easily.

There appears to be no delimiter between bytes when I look at the file. This makes it appear a bit jumbled. Any suggestions?

The "next line" looks like a charcter ">" Should I send 0x3E? If not, what are you using here? Does it matter? Can I use "\n"?

Thanks.

[LT401Vette]

What are you using to view the log?

There is no string output, it is completely binary.
The 1st 81 bytes is the header that is defined in the link above. After that, each record will just be the raw bytes returned by the outpc / och command, the extra 2 bytes are just the block type and a rolling counter.
The point in the block type was to be able to capture different record types, like maybe a trigger logger read, or sound. But the only block type defined or supported right now is the outpc. The other byte is just a rolling counted to help TS find its way back if corruption occurs, but it can just be populated with 0.

So after the 1st 81 bytes it will just be::
[1 byte block type][1 byte rolling counter][ochCommand bytes]

the ochCommand bytes would be the length defined for what ever firmware you are working with. for example 112 bytes for MSII 2.89 or 144 bytes for MS2 Extra.

[jpnovak]

Phil,

I generate a .csv that imports into Excel. Here is a sample of my output taken from the stim. The call is for runtime variables using "a,0,6". the captured format is binary although I can easily switch this to hex as well. The last three are the XYZ axes from an accelerometer. I am also adding GPS coordinates to the mix. (still in development).

0 6 15 160 15 160 17 165 0 251 1 1 128 128 1 1 3 222 3 221 2 188 7 8 2 37 0 115 0 154 0 154 0 246 0 100 0 100 0 100 0 100 0 0 0 100 0 101 0 101 0 172 0 172 0 100 0 0 0 0 0 0 0 27 0 0 3 221 0 100 0 0 0 100 1 19 1 19 0 0 3 221 0 100 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 113 0 0 0 0 17 74 0.054 0.072 0.953


I guess I am still confused about the import compatibility. I can write the format header on top of the file but will TS read this data? My test files with the .frd did not show any delimiter in the file.

In addition, Any suggestions on modifiying the .ini file for additional variables? If the header file in the datalog (.msl) labels a variable will MLV recognize it?

Thanks again for your help.

[LT401Vette]

Where I am confused... What do you mean by binary verse hex? In my thinking, there is no such thing as hex. You can display binary data with an ascii representation of the values in hex, but the data is still binary as in an array of bytes.
TS is not expecting an ascii representation of the data in decimal, binary or hexidecimal. It is looking for the raw bytes as they come back from the controller. Are you actually logging the byte values in ascii?

My test files with the .frd did not show any delimiter in the file.

There is no delimiter. As this is working with raw byte values, you can't really use a delimiter. Any byte value (0-255 or 0-xFF) is a valid value for raw bytes so you can't have a delimiter.

In addition, Any suggestions on modifying the .ini file for additional variables? If the header file in the datalog (.msl) labels a variable will MLV recognize it?

There are 2 ways that you can add variables:

You can create new OutputChannels that are just mathematical functions of the existing defined OutputChannels.
Example :
weirdRPM = rpm +100

To add to the base set of OutputChannels you would need to change the firmware. So you have to work with the data that is coming back from the firmware or change the firmware to send more data.

[jpnovak]

And my lack of understanding has been brought to light.

I see exactly what you are saying. I have a working dash display and I had to work hard to get the data in a character format for display readout. When I first started coding the display project all I got was gibberish in the graphic representation. I bet that gibberish was the actual binary byte output. so yes, I am logging the byte values in ASCII. I converted them.

time to try again.

[jpnovak]

Phil, I am bringing this topic back up. Can you make a suggestion for the data format. Here are my choices (pasted below) using a printf() function. I have been writing as a decimal or integer. I have all of my functionality working except for logging the MS outpc in a recognizable format.

Thank you for your help.

c Character a
d or i Signed decimal integer 392
e Scientific notation (mantise/exponent) using e character 3.9265e+2
E Scientific notation (mantise/exponent) using E character 3.9265E+2
f Decimal floating point 392.65
g Use the shorter of %e or %f 392.65
G Use the shorter of %E or %f 392.65
o Signed octal 610
s String of characters sample
u Unsigned decimal integer 7235
x Unsigned hexadecimal integer 7fa
X Unsigned hexadecimal integer (capital letters) 7FA
p Pointer address B800:0000
n Nothing printed. The argument must be a pointer to a signed int, where the number of characters written so far is stored.
% A % followed by another % character will write % to stdout.

[LT401Vette]

I'm not sure what you are looking to do.
Are you looking at creating a file that has the ASCII representation of the binary values?

That would be a very inefficient way to store and parse the data. What is wrong with just writing the raw bytes instead of the textual value of each byte?

[jpnovak]

There is nothing wrong with using the binary values. I just don't know how to code writing the raw data. I agree its inefficient. However, the step of writing to the SD card from my micro controller is the slow step in the process. I get almost 20Hz when every thing is running.

right now, I poll the MS and load a buffer as an unsigned char. The buffer is then parsed into usable data for an LCD display (my own version of Megaview) and then the entire buffer is written to an SD card. I want to make sure that the data on the SD car can be used by MLV and need some help getting this done.

This is part of a full datalogger development system for my track car. I don't have $3K to buy an off the shelf system datalogger and display. That's also not usually my style. btw, I also have a 3-axis accelerometer and a 10Hz GPS chip loading data at the end of the MS data string. Both of these are connected through an SPI interface to the micro controller and output as a floating decimal character.

Also, I was hoping to add additional channels or variables in MLV to log the acceleration and positioning data with the engine output. I suppose that it might be easier to just log integer values for AFR, RPM, CLT, MAP and a few other key parameters rather than the entire MS output. I can then make my own graphs in a spreadsheet program,

Thanks for your help. I do appreciate your skills.

Jamie

[LT401Vette]

I certainly do appreciate and relate to your do it yourself spirit

I'm thinking, there has to be a simple enough way for you to write the raw data. I'm assuming you are coding in c, but don't know what sort of processor or compiler you have. But even it you are sticking with stdout, what about write or fwrite?

[jpnovak]

Phil,

I will try to see if some of those other functions will work. I am coding in C. I am using a development board using the NXP LPC series microcontroller. Its pretty powerful for its size and has lots of easily configurable serial ports to connect lots of devices. The protoboard has an online compiler that limits some functions but makes it easy to use for a learning programmer like myself. More information here. www.mbed.org

Another question will be... If I am able to write the raw data - any suggestions how to view or otherwise verify that I have the correct format? I have in the past seen giberish on the hyperterminal while running test/debug sequences. I wonder if that was the actual raw data in disguise.

[LT401Vette]

Yep gibberish is usually what raw binary data looks like
A couple of things that will help you read the raw data. If it is in file format, you can download a hex editor. They typically display 16 bytes per row in hex format x00 then will show the ascii values to the right. For read only I often just use TextPad. Whether it displays in Hex format depends on if the file extension being set as a Binary type.

Then I have put a couple helpers in TunerStudio. Under the Tools Menu:
Tools --> "Binary Log Viewer"

It isn't a terribly sophisticated or refined part of TS, but it is helpful.
What it does is similar in that it displays the binary data byte by byte using the ASCII representation in your choice of hex, decimal or bin. But what makes this more helpful is that you can set the wrap size. With the slider, this helps you to line up the records. So for example if each record is 156 bytes long, set the slider on 156 and each records values will be lined up.

There is a bug in the file open, that it limits file extensions to know types. But if in the file name box you type *.* it will show everything.

Another feature in this viewer is to view multiple bytes. If you select 1 to for bytes, dragging from right to left, it will tell you the big endianess and little endianess values of those bytes. Again not very refined as they are really just my own tools, but helpful.

[jpnovak]

I never noticed the binary log viewer before. I think this will help tremendously.

I will report back after I have tried a few ideas for the code.

You have been most helpful. maybe there will be enough interest in this project to do a run of boards.

Best Regards,

jamie

[jpnovak]

Ok I am making progress. If I change the extension of my txt file to .log I can see all the data in the binary viewer. Its all there and lines up perfectly with the slider set to 112.

However, I can not import into TS or MLV. I tried the convert binary log (.frd or .ms3) and this did not work. I suspect I need a header before the file is recognized.

How do I get a header into the file?

Any other suggestions.

[LT401Vette]

112, that sounds right for MSII base code

Now to make it an official frd format it takes a little more data...
That is what is outlined in the frd doc:
http://www.efianalytics.com/TunerStudio ... talog.html

The main thing is the header. which is a fixed 81 bytes long.

Then once you have that written to the file, you write 2 bytes:
x01- The is the block type, x01 is the declares a standard outpc read
Then a rolling counter x00-xFF, but you know what you can just leave this a constant x00
Then the 112 bytes you are reading from the controller

[jpnovak]

I cheated a bit and copied the header line from a frd file created in TS. Here is my 83 byte group. I added line breaks and spaces for clarity.

0x46 0x52 0x44 0x00 0x00 0x0 0x00 0x01 0x4b 0x61
0xf1 0x63 0x4d 0x53 0x49 0x49 0x20 0x52 0x65 0x76
0x20 0x32 0x2e 0x38 0x38 0x30 0x30 0x30 0x20 0x20
0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x51 0x00
0x70 0x01 0x00

The microcontroller is creates an .frd file on the SD card with the header followed by the data.

I still can not get the file conversion to to work. A few things to verify.

Y/N - no line breaks or spaces between bytes for the header
Y/N - no line breaks between the 112 byte outpc group
Can I send these as a hex value?

I feel like I am really close.

[jpnovak]

I am in format heck.

If I send a hex designation it recognizes as two separate bytes. Therefore my header is 166 byes long instead of 83. Yeah, that kinda screws up the offset count. At least I now know what the problem is.

On to find a solution...

[LT401Vette]

Yep taking the cheating approach , just use the first 81 bytes for the header.

Then in a loop do this

start loop
write 2 bytes (x00, x00)
write outpc bytes (the 112)
loop

[jpnovak]

Eureka!

I think you just solved my problem. I didn't realize the two bytes were required on each read cycle loop. I thought they were just tagged in the header...