Go Case Sensitivity With JSON Decoding

TL;DR

• The order of fields in JSON matters when decoding to a Go struct: {"id": 1, "ID": 124452} will map the id as 124452 for a struct with only ID int `json:"id"` defined.
• Explicitly defining the case of the field in the json tag will not necessarily help

Details

About two months ago as of this publish date, one of my coworkers changed some development code which resulted in an interesting break to a monitoring component we have for our data ingest pipeline. The monitoring component is written in Go and sends an event through the data pipeline at specified intervals. Each component in the data pipeline then sends back a message acknowledging it received the message along with some other information. What my coworker ended up changing was moving a piece of transformation code that appended an ID to the JSON event we received (if one did not exist) to run before the monitoring code kicked off. The net effect that this ended up having was transforming this JSON event from to this which broke our monitoring component! We fixed this particular issue by having the monitoring component send out the proper id field but I was curious why this broke our monitoring component.

I ended up playing around with different forms of the JSON that was being sent and how it was decoded to structs with variations of the JSON fields. This is the result of that: which outputs: And here is the Go Playground link.

What does this demonstrate? It shows that ordering of fields with the same name (but different cases) can alter what you expect to receive when unmarshalling JSON. Receiving JSON that looks like {"id": 1, "ID": 124452} can materially effect your program if you are only unmarshalling to a struct with an ID field define like this:

ID int `json:"id"`

Rather than getting 1 you will get 124452 - not exactly what you would expect when you explicitly set the JSON tag to "id" not "ID"!

After tracing through the encoding/json package it ended up that Go's behavior when unmarshalling JSON into structs is use the byte package's EqualFold function when it doesn't find an exact field name match. You can see that here in the encoding/json package's decode.go file. I pulled out the relevant code here: This piece of code is located in the object method on the decodeState struct. What is happening is that for every JSON object this object method iterates through the keys and assigns the data to the appropriate struct field. The code above shows how if there is an exact match for a field name then that field is what the data is assigned to; however, if there is not an exact match then the EqualFold function is employed which is case insensitive. This means that the order in which fields are encountered in the JSON matters as I found in my experiment.

At this time I am not really sure if this would be considered a bug or not. At minimum, I think it's definitely an edge case but one that could cause serious problems.

Update: Here is the link to the Google Group post I made on this.

Go Map Comparisons and JSON Decoding

TL;DR

• Be wary when using reflect.DeepEqual on maps. If there is a chance for a nil map then considering adding the length check when checking for equivalence if your use case calls for it.
• Be aware that when unmarshaling into maps null and {} both result in a nil map

Details

Earlier this week while working on a Go application I was using the internal reflect package to check if two maps were equivalent (using the DeepEqual function). While debugging my code I ran across an interesting situation where the maps I thought were equal (based on fmt.Println statements) were not matching according to reflect.DeepEqual. Here's a decent breakdown of the problem as I initially considered it: What was happening was that I was expecting a JSON object of string-string key-value pairs to be the value for the map key but instead I was occasionally being provided a null value. Based on fmt.Println statements I saw that both paramMap and compMap were "map[]" when this happened. But reflect.DeepEqual still wasn't showing paramMap and compMap were equivalent which in this case was due the operations (JSON decoding) before the reflect comparison. Since the default Go JSON decoder decodes null values to nil the paramMap really had a value of nil and not of an empty map map[]. What then happens is reflect.DeepEqual compares a nil value (hidden by the function parameter specifying a map) to an empty map which you would expect to be equal based on the fmt.Println statements alone; however, that is not the case.

This example code illustrates the confusion a nil map and an empty map can cause which outputs As you can see, a nil map is not equivalent to an empty map despite their string values and lengths being the same. Ultimately, what I decided to do was compare the length of two maps under consideration as well as using reflect.DeepEqual to achieve the desired functionality as shown here (and in the last line of the example code above):

(len(map1) == 0 && len(mapNil) == 0) || reflect.DeepEqual(map1, mapNil)

Here is the interactive example.

While on this topic what do you expect an empty JSON object like {} to decode to when attempting to unmarshal it into map[string]string? An empty map or a nil value? If you guessed an empty map (my original assumption) then you would be wrong. This example code illustrates how a JSON object {} results in a nil map which outputs Here is the interactive example.

Lessons learned:

• Be wary when using reflect.DeepEqual on maps. If there is a chance for a nil map then considering adding the length check when checking for equivalence if your use case calls for it.
• Be aware that when unmarshaling into maps null and {} both result in a nil map