15 Data Files and Serialization

Table constructors provide an interesting alternative for file formats. With a little extra work when writing data, reading becomes trivial. The technique is to write our data file as Lua code that, when run, rebuilds the data into the program. With table constructors, these chunks can look remarkably like a plain data file.

Let us see an example to make things clear. If our data file is in a predefined format, such as CSV (Comma-Separated Values) or XML, we have little choice. However, if we are going to create the file for our own use, we can use Lua constructors as our format. In this format, we represent each data record as a Lua constructor. Instead of writing in our data file something like

      Donald E. Knuth,Literate Programming,CSLI,1992
      Jon Bentley,More Programming Pearls,Addison-Wesley,1990

we write this:

      Entry{"Donald E. Knuth",
            "Literate Programming",
            "CSLI",
            1992}
      
      Entry{"Jon Bentley",
            "More Programming Pearls",
            "Addison-Wesley",
            1990}

Remember that Entry{code} is the same as Entry({code}), that is, a call to some function Entry with a table as its single argument. So, that previous piece of data is a Lua program. To read that file, we only need to run it, with a sensible definition for Entry. For instance, the following program counts the number of entries in a data file:

      local count = 0
      function Entry () count = count + 1 end
      dofile("data")
      print("number of entries: " .. count)

The next program collects in a set the names of all authors found in the file, and then prints them:

      local authors = {}      -- a set to collect authors
      function Entry (b) authors[b[1]] = true end
      dofile("data")
      for name in pairs(authors) do print(name) end

Note the event-driven approach in these program fragments: the function Entry acts as a callback function, which is called during the dofile for each entry in the data file.

When file size is not a big concern, we can use name-value pairs for our representation:^[15]

      Entry{
        author = "Donald E. Knuth",
        title = "Literate Programming",
        publisher = "CSLI",
        year = 1992
      }
      
      Entry{
        author = "Jon Bentley",
        title = "More Programming Pearls",
        year = 1990,
        publisher = "Addison-Wesley",
      }

This format is what we call a self-describing data format, because each piece of data has attached to it a short description of its meaning. Self-describing data are more readable (by humans, at least) than CSV or other compact notations; they are easy to edit by hand, when necessary; and they allow us to make small modifications in the basic format without having to change the data file. For instance, if we add a new field we need only a small change in the reading program, so that it supplies a default value when the field is absent.

With the name-value format, our program to collect authors becomes this:

      local authors = {}      -- a set to collect authors
      function Entry (b) authors[b.author] = true end
      dofile("data")
      for name in pairs(authors) do print(name) end

Now the order of fields is irrelevant. Even if some entries do not have an author, we have to adapt only the function Entry:

      function Entry (b)
        authors[b.author or "unknown"] = true
      end

Frequently we need to serialize some data, that is, to convert the data into a stream of bytes or characters, so that we can save it into a file or send it through a network connection. We can represent serialized data as Lua code in such a way that, when we run the code, it reconstructs the saved values into the reading program.

Usually, if we want to restore the value of a global variable, our chunk will be something like varname = exp, where exp is the Lua code to create the value. The varname is the easy part, so let us see how to write the code that creates a value. For a numeric value, the task is easy:

      function serialize (o)
        if type(o) == "number" then
          io.write(tostring(o))
        else other cases
        end
      end

By writing a float in decimal format, however, we risk losing some precision. We can use a hexadecimal format to avoid this problem. With format ("%a"), the read float will have exactly the same bits of the original one. Moreover, since Lua 5.3 we should distinguish between integers and floats, so that they can be restored with the correct subtype:

      local fmt = {integer = "%d", float = "%a"}
      
      function serialize (o)
        if type(o) == "number" then
          io.write(string.format(fmt[math.type(o)], o))
        else other cases

For a string value, a naive approach would be something like this:

      if type(o) == "string" then
        io.write("'", o, "'")

However, if the string contains special characters (such as quotes or newlines) the resulting code will not be a valid Lua program.

You may be tempted to solve this problem changing quotes:

      if type(o) == "string" then
        io.write("[[", o, "]]")

Beware of code injection! If a malicious user manages to direct your program to save something like " ]]..os.execute('rm *')..[[ " (for instance, she can supply this string as her address), your final chunk will be like this one:

      varname = [[ ]]..os.execute('rm *')..[[ ]]

You will have a bad surprise trying to load this “data”.

A simple way to quote a string in a secure way is with the option "%q" from string.format. This option was designed to save the string in a way that it can be safely read back by Lua. It surrounds the string with double quotes and properly escapes double quotes, newlines, and some other characters inside the string:

      a = 'a "problematic" \\string'
      print(string.format("%q", a))    --> "a \"problematic\" \\string"

Using this feature, our serialize function now looks like this:

      function serialize (o)
        if type(o) == "number" then
          io.write(string.format(fmt[math.type(o)], o))
        elseif type(o) == "string" then
          io.write(string.format("%q", o))
        else other cases
        end
      end

Lua 5.3.3 extended the format option "%q" to work also with numbers (plus nil and Booleans), again writing them in a proper way to be read back by Lua. (In particular, it formats floats in hexadecimal, to ensure full precision.) Thus, since that version, we can simplify and extend serialize even more:

      function serialize (o)
        local t = type(o)
        if t == "number" or t == "string" or t == "boolean" or
           t == "nil" then
          io.write(string.format("%q", o))
        else other cases
        end
      end

Another way to save strings is the notation [=[...]=] for long strings. However, this notation is mainly intended for hand-written code, where we do not want to change a literal string in any way. In automatically generated code, it is easier to escape problematic characters, as the option "%q" from string.format does.

If you nevertheless want to use the long-string notation for automatically generated code, you must take care of some details. The first one is that you must choose a proper number of equals signs. A good proper number is one more than the maximum that appears in the original string. Because strings containing long sequences of equals signs are common (e.g., comments delimiting parts of a source code), we should limit our attention to sequences of equals signs enclosed by square bracket. The second detail is that Lua always ignores a newline at the beginning of a long string; a simple way to avoid this problem is to add always a newline to be ignored.

The function quote in Figure 15.1, “Quoting arbitrary literal strings” is the result of our previous remarks.

      function quote (s)
        -- find maximum length of sequences of equals signs
        local n = -1
        for w in string.gmatch(s, "]=*%f[%]]") do
          n = math.max(n, #w - 1)   -- -1 to remove the ']'
        end
      
        -- produce a string with 'n' plus one equals signs
        local eq = string.rep("=", n + 1)
      
        -- build quoted string
        return string.format(" [%s[\n%s]%s] ", eq, s, eq)
      end

It takes an arbitrary string and returns it formatted as a long string. The call to gmatch creates an iterator to traverse all occurrences of the pattern ’]=*%f[%]]’ (that is, a closing square bracket followed by a sequence of zero or more equals signs followed by a frontier with a closing square bracket) in the string s. For each occurrence, the loop updates n with the maximum number of equals signs so far. After the loop, we use string.rep to replicate an equals sign n + 1 times, which is one more than the maximum occurring in the string. Finally, string.format encloses s with pairs of brackets with the correct number of equals signs in between and adds extra spaces around the quoted string plus a newline at the beginning of the enclosed string.

(We might be tempted to use the simpler pattern ’]=*]’, which does not use a frontier pattern for the second square bracket, but there is a subtlety here. Suppose the subject is "]=]==]". The first match is "]=]". After it, what is left in the string is "==]", and so there is no other match; in the end of the loop, n would be one instead of two. The frontier pattern does not consume the bracket, so that it remains in the subject for the following matches.)

      function serialize (o)
        local t = type(o)
        if t == "number" or t == "string" or t == "boolean" or
           t == "nil" then
          io.write(string.format("%q", o))
        elseif t == "table" then
          io.write("{\n")
          for k,v in pairs(o) do
            io.write("  ", k, " = ")
            serialize(v)
            io.write(",\n")
          end
          io.write("}\n")
        else
          error("cannot serialize a " .. type(o))
        end
      end

      io.write(string.format(" [%s] = ", serialize(k)))

      serialize{a=12, b='Lua', key='another "one"'}

      {
        a = 12,
        b = "Lua",
        key = "another \"one\"",
      }

      {
        ["a"] = 12,
        ["b"] = "Lua",
        ["key"] = "another \"one\"",
      }

      function basicSerialize (o)
        -- assume 'o' is a number or a string
        return string.format("%q", o)
      end
      
      function save (name, value, saved)
        saved = saved or {}                 -- initial value
        io.write(name, " = ")
        if type(value) == "number" or type(value) == "string" then
          io.write(basicSerialize(value), "\n")
        elseif type(value) == "table" then
          if saved[value] then              -- value already saved?
            io.write(saved[value], "\n")    -- use its previous name
          else
            saved[value] = name             -- save name for next time
            io.write("{}\n")                -- create a new table
            for k,v in pairs(value) do      -- save its fields
              k = basicSerialize(k)
              local fname = string.format("%s[%s]", name, k)
              save(fname, v, saved)
            end
          end
        else
          error("cannot save a " .. type(value))
        end
      end

      a = {x=1, y=2; {3,4,5}}
      a[2] = a    -- cycle
      a.z = a[1]  -- shared subtable

      a = {}
      a[1] = {}
      a[1][1] = 3
      a[1][2] = 4
      a[1][3] = 5
      
      a[2] = a
      a["y"] = 2
      a["x"] = 1
      a["z"] = a[1]

      a = {{"one", "two"}, 3}
      b = {k = a[1]}

      local t = {}
      save("a", a, t)
      save("b", b, t)
      
        --> a = {}
        --> a[1] = {}
        --> a[1][1] = "one"
        --> a[1][2] = "two"
        --> a[2] = 3
        --> b = {}
        --> b["k"] = a[1]