Synalysis Q&A - Recent questions in Grammars

Getting started with grammars--a file of simple variable-length records

Thu, 25 Apr 2024 20:15:24 +0000

I'd like to get started learning and using Synalyze It and Hexinator, but I could use a little more help than the docs offer to get me onto the learning curve.

So I'm hoping someone can provide an example grammar to get me started on a simplified version of a file format I'm interested in.

A file in this format consists of a sequence of records, where each record consists of:

An initial 8-bit type value type from a set of predefined values (with a symbolic name for each, but that detail can be left for later).
A 16-bit little endian byteCount of data bytes to follow
A series of byteCount dataBytes
An 8-bit checksum, the two's complement of the sum across type, byteCount, and dataBytes

The full details are much more complex, essentially what you'll see in Intel's OMF-51 object file format specification which you can find at https://www.keil.com/download/files/omf51.zip, but I'm happy to put off all of that until I know how to write a suitable grammar for the basic format.

A script to check a record's checksum would also be helpful but can be left for later.

how do i reject a zero length string - or alternatively how do i terminate parsing an element from a script?

Wed, 24 May 2023 02:39:39 +0000

Hi,

I am trying to write a grammar for EXR files. These have a header with a list of attributes. each attribute consists of an attribute name (string), type (another string), size (int) and then some data (dependent on the size). There are various predefined attribute types that I can write grammars to parse, and can fall back to a base-case undefined attribute for any unknown attribute types. This seems to be mostly working.

The problem is that this list of attributes is null terminated. So I need to know when there is a zero at the start of the next structure, and then move on to parsing the next bit of the file. Since the grammar is trying to parse another attribute, this shows up as a zero length attribute name (attribute names are zero-terminated). As far as I can tell there is no way to reject a zero-length string. And no direct way to detect the null terminator. I have written a script that detects the zero-length string, but then I don't know how to control the parser to tell it to move on. Any ideas?

thanks

What does the "Consists of" box do exactly?

Thu, 27 Apr 2023 15:37:58 +0000

I've been trying to figure what this actually does. In experimenting with it, I irrecoverably lost a whole grammar I was working on, so I'll just ask you.

Grammer Editor

Tue, 07 Mar 2023 06:30:08 +0000

I have a file the first block starts at 0 and is 512 bytes long

The 2nd block starts at 512 and is 512 bytes long.

I cannot figure out how to set the offset for the 2nd block

Volume

LBN 1

LBN 2

I not having a problem defining the structure for LBN2

How do I start using grammar?

Thu, 16 Feb 2023 06:48:56 +0000

I recently discovered Hexinator and I would like to try out the grammar feature, just to see how it works. I downloaded the WAV file grammar from the grammars page (https://www.synalysis.net/formats.xml) and now I have wav.grammar file with some XML code. I also opened a small wav file in Hexinator and I believe the program is supposed to show the different parts of the WAV header and the data sections in different colours. But I have no idea how to get started. Could someone please link me to a tutorial or a how to guide with screenshots? Or is this a paid only feature? I am using the free version in Windows 10.

Can the number of repetitions of a structure be set by a number from within another structure?

Wed, 23 Nov 2022 05:15:21 +0000

Specifically, I've got a structure that's a fixed size data block, that's meant to repeat exactly the number of times specified by a number from another structure earlier in the file. There's the repeat dropdown, that seems like it might be intended for this sort of purpose, but the only option in it is "<none>", and the min and max values don't seem to have any way to fill in a variable either.

How to cast a data type?

Wed, 09 Feb 2022 17:04:51 +0000

The file format I'm working with (NITF) stores the record count as ASCII to represent a 3-digit number between 000-999. Is there a way to transform that value to the corresponding integer so I can use it to repeat the record structure?

Has someone generated a grammar for m-bus (meter bus) yet?

Thu, 02 Dec 2021 16:31:43 +0000

I'd like to parse a hex file with a dump of a m-bus protocol. Has someone prepared the grammar for analyzing m-bus protocols? That would be very nice.

Thank's a lot.

Powerman

Is there a way to get nibble arrays to obey endianness?

Wed, 04 Aug 2021 14:00:53 +0000

I'm working on a grammar where there is an array of 4-bit bitfields. There are 1000 entries, where each entry has 4 single-bit flags. They are stored in a little-endian fashion: each byte stores a pair of entries, where the four least significant bits are the first entry and the most significant bits are the second.

Since it is an array, there are two structs involved: an inner struct representing a single 4-bit entry, and an outer struct containing the 1000 repetitions of the inner struct.

The problem is, even if I set both of these structs to little endian, each pair of nibbles is parsing out of order. That is, entries 0 and 1 are swapped, 2 and 3 are swapped, 4 and 5 are swapped, etc.

Is there a way to make Hexinator handle this properly?

Can a grammar reference another grammar?

Thu, 03 Sep 2020 00:12:27 +0000

For instance I would like to create a grammar for ID3 tags which could be used in both flac and mp3 grammars

Conditional grammar structures

Mon, 13 Jul 2020 23:09:41 +0000

I'm writing a grammar for MPLS files.

One of the elements is StreamType (1 byte numeric), and if StreamType is 1 it is followed by by the A structure only. If StreamType is 2 it is followed by the B, C, and A structures in that order, and if StreamType is 3 it is followed by the B and A structures in that order.

How do I accomplish this?

I don't have the pro-version, so no scripting as I understand it unfortunately.

Value field of a structure

Fri, 28 Feb 2020 05:56:58 +0000

Short version:

Can "Value," on a structure, only be the name of a field inside?

Long version:

I've got some bytes that are essentially a keypath. As an example:

81 A4 74 68 69 73 A2 69 73 A2 6D 79 A7 6B 65 79 70 61 74 68 | ..this.is.my.keypath

Byte 0 is just a type specifier, indicating what kind of binary value would be found after this keypath. Bytes 1, 6, 9, and 12 are length indicators of the form A<length>. Byte 1 is A4, so the first word (this) is 4 bytes long. My grammar is (simplified):

<structure name="keypath" valueexpression="key">
    <structure name="key" valueexpression="value">
        <binary name="magic" length="4" lengthunit="bit"/>
        <binary name="length" length="4" lengthunit="bit"/>
        <string name="value" type="fixed-length" length="length"/>
    </structure>
</structure>

The value of keypath shows as the first instance of value, or this. I want the value of keypath to show something like this|is|my|keypath but all I get is this. Is there something I can set the value expression to that will do this for me, or am I stuck writing a script?

Synalize It only recognizes first instance of element in custom grammar?

Tue, 26 Nov 2019 05:47:14 +0000

Hi, I'm trying to use Synalize It to help me design a file format for a college project, but have the issue that it only recognizes the first instance of the data, and concludes the rest of the file is just padding. I have repeat settings set correctly (I think), but can't figure out the cause of the problem! Any help or suggestions are appreciated, Thanks!

Here's the contents of the grammar file:
"<?xml version="1.0" encoding="UTF-8"?>
<ufwb version="1.17">
   <grammar name="SmartShelf Metadata" start="id:197" author="Patrick L">
       <description>Grammar for the Mech 90 SmartShelf Metadata</description>
       <structure name="BinContainer" id="213" encoding="ISO_8859-1:1987" endian="big" signed="no" order="variable">
           <structure name="Bin" id="197" length="64" alignment="1" repeatmax="-1" encoding="ISO_8859-1:1987" endian="big" signed="no">
               <binary name="Magic_Number" mustmatch="yes" id="198" strokecolor="FF0000" length="1">
                   <fixedvalues>
                       <fixedvalue name="Magic Number" value="B1"/>
                   </fixedvalues>
               </binary>
               <number name="Bin_ID" id="199" fillcolor="0000FF" type="integer" length="1"/>
               <number name="Offset_Steps" id="200" strokecolor="0000FF" type="integer" length="1"/>
               <number name="Steps_From_Home" id="201" strokecolor="00F1FF" type="integer" length="2"/>
               <number name="Protocol_Version" id="202" type="integer" length="1">
                   <fixedvalues>
                       <fixedvalue name="V1" value="1"/>
                   </fixedvalues>
               </number>
               <number name="<reserved>" id="203" strokecolor="999999" repeatmin="4" repeatmax="4" type="integer" length="1"/>
               <string name="Bin_Name" id="204" fillcolor="21D2C4" type="fixed-length" length="20"/>
               <string name="Bin_Description" id="205" fillcolor="A2FFFF" type="fixed-length" length="30"/>
               <binary name="Padding" mustmatch="yes" id="206" repeatmin="0" repeatmax="-1" length="1">
                   <fixedvalues>
                       <fixedvalue name="Padding" value="77"/>
                   </fixedvalues>
               </binary>
           </structure>
           <structure name="ExtendedBinData" id="208" length="64" disabled="yes" alignment="0" repeatmax="-1" encoding="ISO_8859-1:1987" endian="big" signed="no">
               <binary name="Magic_Number" id="209" length="remaining">
                   <fixedvalues>
                       <fixedvalue name="Magic_Number" value="B3"/>
                   </fixedvalues>
               </binary>
               <number name="Bin_ID" id="210" type="integer" length="1"/>
               <number name="<reserved>" id="211" repeatmax="-1" type="integer" length="1"/>
           </structure>
       </structure>
   </grammar>
</ufwb>
",
and here's enough my test file, consisting of two identical "Bin" Elements, and padded with 0xFFs:
"B1012000200100000000534D4420636F6D706F6E656E74732E000000000053757266616365204D6F756E74205265736973746F72732C20657463000077777777B1012000200100000000534D4420636F6D706F6E656E74732E000000000053757266616365204D6F756E74205265736973746F72732C20657463000077777777FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"

What is the best way to add a binary coded decimal (BCD) number to a grammar?

Tue, 27 Nov 2018 06:14:43 +0000

What would be the best way to add a binary coded decimal number to a grammar? The best way I can come up with is to create a 4-bit number that's limited to a max value of 9 and repeat it. Is there a better way? Would I have to resort to scripting if I want the multi-digit number to appear as one element?

Upper half of 8-byte le uint constants not correctly loaded from grammar

Fri, 03 Nov 2017 22:47:49 +0000

I'm trying to build tools for a file format with lots of 64-bit little-endian unsigned integers as constants. Even when they're correctly represented in an on-disk .grammar file, when that file is loaded, many of the constants take on the specific value "0x7FFFFFFFFFFFFFFF".

It looks like perhaps 0x8000000000000000 is the largest constant value which is correctly managed?

How to adjust the length and alignment of a structure dynamically

Thu, 31 Aug 2017 10:07:41 +0000

I'm analysing a binary file that can be divided in mainly three sections (Head, body and footer). The head and footer are both 36 bytes long and the body expends its length dynamically, depending on what is stored in the file. Now I want to have a gramma that recognizes how long the body structure is. Also I need to adjust the alignment of the footer structure, so that the last 36 byte of the file are included in this structure element. The length of the file - (36 bytes head + 36 bytes footer) = lengthe of the body structure. I tryed a long time to find a solution for this problem but without a satisfactory success.

Can a structure definition call itself?

Tue, 07 Feb 2017 05:50:46 +0000

I have a file with nested repeating structures. I need to use a count field to repeat a structure which may contain a nested structure of the same type. Does Synalysis handle this recursion, given that the repeat count value will have the same name with different values in the two nested instances?

Variable length determined by first bit of a byte

Mon, 23 Jan 2017 22:28:05 +0000

I'm creating a grammar for MIDI-files.

One structure is a variable length number field where the first bit indicates if this is the last byte of data in the structure and the remaining 7 bits are actual data. As long as the first bit is 1 the next byte is also a part of the number field.

Examples:

0100 0000 = 100 0000
10100 0000 1100 1100 0100 0000 = 0 1000 0010 0110 0100 0000

How do I accomplish this in Synalyze it!?

Referring to a Data Type script from a script element

Sun, 18 Sep 2016 00:01:31 +0000

Is there any good way of referring to a data type script from a script element? For example I have a DataType script to parse varInts and I would like to use it as part of parsing a result in a script element.

Mapping a variable length value in an offset

Sat, 17 Sep 2016 23:55:18 +0000

So I have a grammar that has a root level object in it with an offset.

The offset refers to a structure of type "Values" that contains

a: custom value (a varint from an earlier question)
b: a number
c: a script

The script takes the number as an enumeration for figuring out what the rest of foo is:

function numberfunc(element, byteView, offset, results, length)
number = byteView:readUnsignedInt(offset, length, synalysis.ENDIAN_LITTLE)
value = synalysis.Value()
value:setName("Value")
value:setUnsigned(number)
results:addElement(element, length, 1, value)
end

function floatingfunc(element, byteView, offset, results, length)
number = byteView:readUnsignedIntBits(offset * 8, length * 8, synalysis.ENDIAN_LITTLE)
value = synalysis.Value()
value:setFloat(number)
-- add element with value to results
results:addElement(element, length, 1, value)
end

function int8func(element, byteView, offset, results)
numberfunc(element, byteView, offset, results, 1)
end

function int16func(element, byteView, offset, results)
numberfunc(element, byteView, offset, results, 2)
end

function int32func(element, byteView, offset, results)
numberfunc(element, byteView, offset, results, 4)
end

function int64func(element, byteView, offset, results)
numberfunc(element, byteView, offset, results, 8)
end

function truefunc(element, byteView, offset, results)
value = synalysis.Value()
value:setString("true")
results:addElement(element, 0, 0, value)
end

function falsefunc(element, byteView, offset, results)
value = synalysis.Value()
value:setString("false")
results:addElement(element, 0, 0, value)
end

function floatfunc(element, byteView, offset, results)
floatingfunc(element, byteView, offset, results, 4)
end

function doublefunc(element, byteView, offset, results)
floatingfunc(element, byteView, offset, results, 8)
end

function datafunc(element, byteView, offset, results)
end

function nilfunc(element, byteView, offset, results)
end

function objectfunc(element, byteView, offset, results)
number = byteView:readUnsignedInt(offset, 4, synalysis.ENDIAN_LITTLE)
value = synalysis.Value()
value:setUnsigned(number)
value:setName("ObjectIndex")
element:setLength(4, synalysis.LENGTH_UNIT_BYTES)
results:addElement(element, 4, 1, value)
end

-- get collection with results so far
results = currentMapper:getCurrentResults()
-- get latest added result
lastResult = results:getLastResult()
-- access the parsed value
value = lastResult:getValue()
-- get the value parsed
num = value:getUnsignedNumber()

local funcs = {
int8func,
int16func,
int32func,
int64func,
truefunc,
falsefunc,
floatfunc,
doublefunc,
datafunc,
nilfunc,
objectfunc
}

element = currentMapper:getCurrentStructureElement()
byteView = currentMapper:getCurrentByteView()
offset = currentMapper:getCurrentOffset()

funcs[num + 1](element, byteView, offset, results)

datafunc and nilfunc aren't implemented yet (and please excuse my Lua, I'm just learning).

Anyways, once I parse my file the values being read in by my script don't appear to be being added to my offsets.

I would expect Values[1] to be being read in from offset 2031 instead of 2027. What am I missing? I've tried setting the currentOffset amongst other things with no luck.

String ending in \r\n

Wed, 17 Aug 2016 00:36:23 +0000

Hello!

I've tried to analyze capture file from http response and I'm wondering if it's possible to add to grammar strings ending in \r\n or "0D0A" characters. When I try to use delimited string ofcourse I can't type these characters and using hex codes doesn't work either. So is it possible or maybe I should use another method to do this?

Best regards!

Mariusz Cegiełka