title: How to store binary data in a Lua string
body: <p>I needed to create a custom file format with embedded meta information. Instead of whipping up my own format I decide to just use Lua.</p>
  <pre><code>texture
  {
     format=GL_LUMINANCE_ALPHA;
     type=GL_UNSIGNED_BYTE;
     width=256;
     height=128;
     pixels=[[
  &lt;binary-data-here&gt;]];
  }
  </code></pre>
  <p><code>texture</code> is a function that takes a table as its sole argument. It then looks up the various parameters by name in the table and forwards the call on to a C++ routine. Nothing out of the ordinary I hope.</p>
  <p>Occasionally the files fail to parse with the following error:</p>
  <pre><code>my_file.lua:8: unexpected symbol near ']'
  </code></pre>
  <p>What's going on here?<br />
  Is there a better way to store binary data in Lua?</p>
  <hr />
  <h3>Update</h3>
  <p>It turns out that storing binary data is a Lua string is <a href="http://batbytes.com/luafaq/#T1.39" rel="nofollow noreferrer">non-trivial</a>. But it is possible when taking care with 3 sequences.</p>
  <ul>
  <li><p><a href="http://www.lua.org/manual/5.2/manual.html#3.1" rel="nofollow noreferrer">Long-format-string-literals</a> cannot have an embedded <em>closing-long-bracket</em> (<code>]]</code>, <code>]=]</code>, etc).<br />
  This one is pretty obvious.</p>
  </li>
  <li><p>Long-format-string-literals cannot end with something like <code>]==</code> which would match the chosen <em>closing-long-bracket</em>.<br />
  This one is more subtle. Luckily the script will fail to compile if done wrong.</p>
  </li>
  <li><p>The data cannot embed <code>\n</code> or <code>\r</code>.<br />
  Lua's <a href="http://www.lua.org/source/5.1/llex.c.html#read_long_string" rel="nofollow noreferrer">built in line-end processing</a> messes these up.  This problem is much more subtle.  The script will compile fine but it will yield the wrong data. 0x13 =&gt; 0x10, 0x1013 =&gt; 0x10, etc.</p>
  </li>
  </ul>
  <p>To get around these limitations I split the binary data up on <code>\r</code>, <code>\n</code>, then pick a <em>long-bracket</em> that works, finally emit Lua that concats the various parts back together.  I used a script that does this for me.<br />
  input: <code>XXXX\nXX]]XX\r\nXX]]XX]=</code></p>
  <pre><code>texture
  {
    --other fields omitted      
    pixels= '' ..
       [[XXXX]] ..
       '\n' ..
       [=[XX]]XX]=] ..
       '\r\n' ..
       [==[XX]]XX]=]==];
  }
  </code></pre>