Giving QR Codes Superpowers

QR codes are popular because they can store more information than bar codes, but they can’t store binary data and have no inherent structure. In this post, I’ll show how to overcome these limitations and give QR codes superpowers!

QR Codes and Their Limitations

QR codes were invented by the Japanese company Denso Wave in the mid-90s as a way to get higher information density in machine-readable optical labels. They support a number of encoding methods including numeric digits, alphanumerics, kanji, and raw text. Although the QR format does have an encoding called “byte mode”, each byte in this mode represents an ISO 8859-1 character, not binary data.

ECI escape sequences allow encoding of nonstandard text encodings, but each escape code bloats the message with out-of-band data (they have to be long in order to minimize the chance of misinterpreting data as escapes), and still binary data is not supported 😞

If you did try to write binary data, it would either be rejected (if the decoder is conformant), or just end up garbled once a reader tries to interpret it as text (some decoders apply heuristics to guess at the intended format, complicating things even further). So there’s no reliable way to directly encode binary data into a QR code, let alone ad-hoc structured binary data.

However, with a little creative thinking, some minor algorithmic tweaks and the Concise Encoding format , it’s possible to work around this limitation to reliably read and write binary ad-hoc structured data in QR codes for slightly modified decoders!

To see how, let’s take a look at the text encoding used for “byte mode”: ISO 8859-1.

Notice how the codes 7f-9f are unassigned (many text formats do this for legacy reasons ). This means that any non-ECI data containing these values is invalid because it can’t be decoded as ISO 8859-1 text.

Side Note: Non-conformant QR decoders might not actually validate this, which is why you sometimes get strange garbled results after decoding a QR code that was mistakenly encoded in UTF-8 without an ECI header (by a non-conformant encoder).

So even though there are no technical limitations against using them, unassigned characters are considered invalid, which means that any valid QR code will not contain them.

So, what we could do is re-purpose one of these unassigned characters and use it as a sentinel to indicate the presence of special, non-textual data. When a scanner that knows about the sentinel byte encounters it as the first byte of the payload, it can switch decoding modes reliably.

Existing standards-compliant QR decoders will still function correctly because they’ll rightly reject the code due to the (technically invalid) sentinel byte. They won’t be able to decode the binary data, but they also won’t malfunction and give incorrect results.

Encoding ad-hoc binary data into QR codes

Concise Encoding is an ad-hoc data format with a text and a binary encoding form. In the binary format (CBE) , all documents begin with the sentinel byte 0x81 (specifically chosen because it’s an invalid starting byte in most popular text formats, including ISO 8859 and UTF-8). We’ll leverage this to encode a CBE document into a QR code.

I’ve adapted https://github.com/kstenerud/enctool to support QR codes and initiate special processing when the first byte of the QR data is 0x81. You can follow along by installing the go language on your system and then installing enctool like so:

1go install github.com/kstenerud/enctool@latest

This will install enctool into $GOPATH/bin (usually go/bin in your home directory).

We’ll be using enctool’s convert functionality. You can see the available options by typing:

1enctool convert -h

Example

As an example, let’s say that we’re developing a new “international shipping and storage requirements” QR code format.

Here’s a sample document that we want to encode into a QR code. I’m writing it here in the text format (CTE ) for convenience, but the actual data will be written in the binary format (CBE ).

 1c1 {
 2    "temperature range" = [-20 5]
 3    "hazards" = [
 4        "pressurized"
 5        "flammable"
 6        "fragile"
 7    ]
 8    "max tilt degrees" = 15
 9    "perishes after" = 2022-12-05
10}

Save this document as with-text.cte, and then convert to CBE like so:

1enctool convert -s with-text.cte -sf cte -df cbe -d with-text.cbe

How big is it?

1ls -l with-text.cbe
2-rw-rw-r-- 1 karl karl 105 Jan  6 17:15 with-text.cbe

A whopping 105 bytes (mainly due to the many text fields the data contains). We could encode it as-is, but that would produce a pretty big QR code.

What if instead we came up with a schema that replaces all well-known text keys and values with integer enumerations? That should really shrink things!

Note: For completeness sake we’ll also include a fourCC style identifier so that any reader can identify which schema and version the data was encoded with (let’s say that key 0 is always used to specify the schema).

Fictional Schema:

• 0 = schema ID and version adherence: (fourCC-style integer: “TSS” + version)
• 1 = temperature range: (list of two integers)
• 2 = hazards: (list of enumerated integers):
  • 0 = corrosive
  • 1 = photoreactive
  • …
  • 4 = pressurized
  • …
  • 6 = flammable
  • …
  • 19 = fragile
• …
• 4 = max tilt degrees: (integer)
• …
• 9 = perishes after: (date)

Document (CTE):

 1c1 {
 2    0 = 0x54535301 // Transport and Storage Schema version 1 ("TSS" + 1)
 3    1 = [-20 5]    // Temperature range from -20 to 5
 4    2 = [          // Hazards:
 5        4             // Pressurized
 6        6             // Flammable
 7        19            // Fragile
 8    ]
 9    4 = 15         // Max 15 degrees tilt
10    9 = 2022-12-05 // Perishes after Dec 5, 2022
11}

Because integers from -100 to 100 are their own type codes in CBE (thereby encoding into a single byte), you can achieve tremendous space savings by using them as enumerated types.

Let’s see how our modifications do (save this document as with-enums.cte):

1enctool convert -s with-enums.cte -sf cte -df cbe -d with-enums.cbe

Now how big is it?

1ls -l with-enums.cbe 
2-rw-rw-r-- 1 karl karl 28 Jan  6 17:22 with-enums.cbe

We’ve shrunk down our payload from 105 bytes to 28 bytes! Much better!

Here are the byte contents of the CBE document. Note the first byte 0x81, which is an unassigned character in ISO 8859-1:

181 00 99 00 6c 01 53 53 54 01 9a ec 05 9b 02 9a 04 06 13 9b 04 0f 09 7a 85 59 00 9b

Now let’s encode this data into a QR code (border size 1, 400x400 pixels):

1enctool convert -s with-enums.cte -df qr -d qr.png -b 1 -is 400

This creates qr.png, which contains our QR code:

Now let’s read the data back. I’ll convert it to CTE for convenience, printing to stdout:

1enctool convert -s qr.png -sf qr -df cte

Output:

 1c0
 2{
 3    0 = 1414746881
 4    1 = [
 5        -20
 6        5
 7    ]
 8    2 = [
 9        4
10        6
11        19
12    ]
13    4 = 15
14    9 = 2022-12-05
15}

Once we’ve got our document, we can process it through the schema to recover the true meaning of the data.

So there you have it: Supercharged QR codes that can hold arbitrary structured data! Neat!