Announcing KeyVal 1.0.0

At every company I've worked for, we've had the very simple problem of storing work-area parameters that we've solved with completely custom reinventions of the wheel.  At Anonymous Company One, for example, we extended INI format:

[local]
library=mylib
design=mydesign
optimization_level=2

At Anonymous Company Two, we defined an XML schema:

<config>
  <local>
    <var>
      <name>library</name>
      <value>mylib</value>
    </var>
    <var>
      <name>design</name>
      <value>mydesign</name>
    </var>
    <var>
      <name>optimization_level</name>
      <value>2</value>
    </var>
  </local>
</config>

In both cases, the settings defined by these files are identified to consumers by the following paths:

• local::library
• local::design
• local::optimization_level

Both solutions allowed us to embed variables into values, so we had code that would figure out how to interpolate values before giving them to the user.

Due to the wheel-reinventing, both of these required a good deal of developer overhead.  The INI solution was implemented with a flex-generated parser, which meant updating the code any time users found something that broke flex's assumptions.  (My favorite was when emacs would write out a file whose last character was not a carriage return -- since you can't put <<EOF>> at the end of a pattern, we couldn't treat it like a carriage return, and the last setting of the file would be dropped.) The XML solution came with a standard parser, but we still had to pick the data out. (If you'll notice, the <var> sections above actually consist of "whitespace + <name> section + whitespace + <value> section + whitespace" -- all of that whitespace is considered data by XML parsers, so we had to selectively toss data fields. Also, notice that we had to strip out the <var> level of hierarchy!)

My worst gripe with both those solutions, however, is that it's not straightforward to find things from the commandline.  For developers debugging some user's work area, one of the most common queries is what some key is set to.  grep fails us here.  For INI, we can look for the key name, or we can look for the section name, but not both, and we need both.  For XML, we can look for the key name, but the value's on a different line (maybe the one before, maybe the one after, or maybe there's other stuff in that section), and finding the rest of the XML path in a real-world configurations file is a frustrating exercise in visual parsing.

There are other solutions such as JSON and YAML, but, like XML, they too have flexible whitespacing and ordering that prevents commandline access from being straightforward.

What we really need is a config-file format with greppable fields that comes with an open-source, rock-solid parser and API.  What we really need is KeyVal.

KeyVal's file format lists every full path and value on its own line:

`local::library` = `mylibrary`
`local::design` = `mydesign`
`local::optimization_level` = `2`

This format is very rigid - you always have to specify the full key path, and the value always has to be on the same line.  (You can put spaces around the '=' all you like though.)  You cannot put two key=value pairs on a single line.

The KeyVal parser was not generated by flex but rather is a full custom parser written in C.  The database part does keep all settings in memory, but to ensure scalability it uses only as much as the input you give it.  The KeyVal API natively supports variables in values.  KeyVal was designed to be the best tool to use for configurations files.

For non-C users, the KeyVal package comes with a SWIG control file.  I have created real object-oriented interfaces for perl, python, and tcl.  Included in the package is a Makefile.swig that will get you 90% of the way toward building those.  (I say "90%" because cross-language bindings on arbitrary platforms is not a solved problem.  It works on my mac, and with a little massaging works on RedHat 5.  SWIG is notoriously fickle.)

KeyVal's full documentation and source code is available on GitHub:

https://github.com/CheetoMonster/KeyVal

The direct link to download the 1.0.0 version is here.