Update on compression for near-consecutive or grouped tabular data

Development of Track is now mostly stagnant whilst my priorities lie with different projects and uni work. However that hasn’t stopped me from wanting to improve the compression algorithm outlined in the first post, and address some of the things I initially forgot to mention.

Variables

In the first post, the links to examples contained variables, however I hadn’t documented how they work. The variables are written at the top of the file, with one line representing each definition. Think of this as a header, all of the variables that will be used throughout the decompression process are expressed here.

AWR
LWR
~

Note that the tilda ~ symbol denotes the ending of the variables section, and that any content hereafter will not be interpreted as a variable.

From this above, 3 variables have been created and can be accessed with a dollar symbol $ and their relative index (as if they are represented in a list). For example:

$0 -> AWR
$1 -> LWR

Variables can be accessed inside of the main body, as such:

453101 | $0 | 8 | 2010
+      | /  | / | +
+      | /  | 5 | /
+      | $1 | / | /
+      | /  | / | /

Concatenation

Variables can also be concatenated with each other using a semi-colon ;, to denote a new segment. Using a variable within a wider value is easy, just access the variable with a dollar symbol $ where it is needed. For example:

$0; $1; -> AWR LWR
$0; and $1; -> AWR and LWR
$0; tomatoes $1; carrots -> AWR tomatoes LWR carrots

Shortcuts

To be honest, I’m not entirely sure what to name this section so shortcuts will have to do. In the example files, I used a technique not documented in the original post, that involves leaving out as much repeated data as possible (hence why it’s called shortcuts).

In the example below, everything after the first column is the same as the record above it, so everthing after the data that needs to change is just left out!

1 | foo | bar
+
+

During the decompression process, everything left out just filled with the values from before. This represents:

1 | foo | bar
2 | foo | bar
3 | foo | bar

Conclusion

This is just a short update to clarify a few things about the original post. There’s currently no real process out there for compressing and decompressing data in this format, so I might work on something like that soon. I’ll probably make another post with more clarifications and newer features.