Extension

English | 中文

json-iterator provides some options for serialization/deserialization, but they cannot cover complicated use case. For example, there's not any option json-iterator has provided can help if you want your boolean values to be encoded as integer. However, json-iterator provide an Extension mechanism for us to customize the serialization/deserialization

ValEncoder/ValDecoder interface

Before introducing the use of Extension, we need to know what is ValEncoder and ValDecoder in json-iterator. When we use Extension, we are actually registering different ValEncoder and ValDecoder for different types. Please note that ValEncoder/ValDecoder is different from json.Encoder/json.Decoder, don't confuse them.

• ValEncoder

  type ValEncoder interface {
      IsEmpty(ptr unsafe.Pointer) bool
      Encode(ptr unsafe.Pointer, stream *Stream)
  }

  ValEncoder is an encoder used by json-iterator to encode a certain type of data. Its two member methods are introduced as bellow:

  • Encode

    Encode function is used to encode certain types of data. ptr points to the value to be encoded. stream provides different methods for users to write various types of data to the output stream. When you implement ValEncoder, what you need to do in this function is to convert ptr into a pointer of the data type corresponding to this ValEncoder, and then call the stream's methods to encode and output the value pointed to by ptr

  • IsEmpty

    IsEmpty is a function related to the option omitempty(check godoc of encoding/json for more information) of json struct field tag. If you're an encoding/json user, you may know that if a struct field's omitempty option of json tag is enabled, this field will be omitted when encoding into JSON if the value of this filed "is empty". IsEmpty is a function for you to define what means "is empty" with your value. Therefore, what you need to do in this function is to convert ptr into a pointer of the data type corresponding to this ValEncoder and check the value if it's empty

  Let's look into an example to help us to understand ValEncoder. json-iterator provides a built-in TimeAsInt64Codec, here's its implementation:

  func (codec *timeAsInt64Codec) IsEmpty(ptr unsafe.Pointer) bool {
      ts := *((*time.Time)(ptr))
      return ts.UnixNano() == 0
  }
  
  func (codec *timeAsInt64Codec) Encode(ptr unsafe.Pointer, stream *jsoniter.Stream) {
      ts := *((*time.Time)(ptr))
      stream.WriteInt64(ts.UnixNano() / codec.precision.Nanoseconds())
  }

  In the Encode function, ptr is converted into a pointer pointing to time.Time type, and then dereferenced to get the time.Time object pointed to by it. Next, call its member function to calculate its corresponding unix time, and finally call the WriteInt64 function provided by stream to encode and output the int64 type unix time.

  IsEmpty gets the time.Time object pointed to by ptr in the same way, and then regard the value of the object as empty if its converted unix time is 0

• ValDecoder

  type ValDecoder interface {
      Decode(ptr unsafe.Pointer, iter *Iterator)
  }

  ValDecoder is a decoder used by json-iterator to decode a certain type of data. Its member method is introduced as bellow:

  • Decode

    Decode function is used to decode a certain type of data. ptr points to the value to be set after decoding. iter provides different methods for users to read various types of data from the input stream. When you implement ValDecoder, what you need to do in this function is to call iter's method to read the data from input stream, and then convert ptr into a pointer of the data type corresponding to this ValDecoder, and set the value pointed by ptr with the data we read from input stream by iter

  Again look into the example of TimeAsInt64Codec

  func (codec *timeAsInt64Codec) Decode(ptr unsafe.Pointer, iter *jsoniter.Iterator) {
      nanoseconds := iter.ReadInt64() * codec.precision.Nanoseconds()
      *((*time.Time)(ptr)) = time.Unix(0, nanoseconds)
  }

  Decode function call iter's ReadInt64 method to read a int64 type value from the input json stream. Next, because the data type corresponding to our ValDecoder is time.Time, ptr is converted to a pointer pointed to time.Time type, and the value it pointed to is set to a time.Time object created with the int64 value read from the input json stream by iter as the Unix Time. In this way, we have completed the function that read an int64 type unix time from the json string and deserialize it into a time.Time object

Customize your extension

To customize the serialization/deserialization extension, you need to implement the Extension interface and register it by calling RegisterExtension, which contains the following methods:

type Extension interface {
    UpdateStructDescriptor(structDescriptor *StructDescriptor)
    CreateMapKeyDecoder(typ reflect2.Type) ValDecoder
    CreateMapKeyEncoder(typ reflect2.Type) ValEncoder
    CreateDecoder(typ reflect2.Type) ValDecoder
    CreateEncoder(typ reflect2.Type) ValEncoder
    DecorateDecoder(typ reflect2.Type, decoder ValDecoder) ValDecoder
    DecorateEncoder(typ reflect2.Type, encoder ValEncoder) ValEncoder
}

There is a DummyExtension in json-iterator, which is a basic implementation of Extension (basically do nothing or return empty). When you are defining your own Extension, you can anonymously embed DummyExtension, so you don’t need to implement all the Extension members, just focus on the functions you need. Below we use some examples to illustrate what each member function of Extension can be used for

• UpdateStructDescriptor

  In the UpdateStructDescriptor function, we can customize the encoder/decoder of a field of the structure, or control which strings are bound when the field is serialized/deserialized

  type testCodec struct{
  }
  
  func (codec *testCodec) Encode(ptr unsafe.Pointer, stream *jsoniter.Stream){
      str := *((*string)(ptr))
      stream.WriteString("TestPrefix_" + str)
  }
  
  func (codec *testCodec) IsEmpty(ptr unsafe.Pointer) bool {
      str := *((*string)(ptr))
      return str == ""
  }
  
  type sampleExtension struct {
      jsoniter.DummyExtension
  }
  
  func (e *sampleExtension) UpdateStructDescriptor(structDescriptor *jsoniter.StructDescriptor) {
      if structDescriptor.Type.String() != "main.testStruct" {
          return
      }
  
      binding := structDescriptor.GetField("TestField")
      binding.Encoder = &testCodec{}
      binding.FromNames = []string{"TestField", "Test_Field", "Test-Field"}
  }
  
  func extensionTest(){
      type testStruct struct {
          TestField string
      }
  
      t := testStruct{"fieldValue"}
      jsoniter.RegisterExtension(&sampleExtension{})
      s, _ := jsoniter.MarshalToString(t)
      fmt.Println(s)
      // Output:
      // {"TestField":"TestPrefix_fieldValue"}
  
      jsoniter.UnmarshalFromString(`{"Test-Field":"bbb"}`, &t)
      fmt.Println(t.TestField)
      // Output:
      // bbb
  }

  In the above example, first we implemented a ValEncoder with testCodec, which added a "TestPrefix_" prefix in front of the string when encoding it. Then we registered a sampleExtension. In the UpdateStructDescriptor function, we set the encoder of TestField field of testStruct to the testCodec, and then bound it with several alias strings. The effect obtained is that when this structure is serialized, the contents of TestField will be prefixed with "TestPrefix_ ". When deserialized, the alias of TestField will be mapped to this field

• CreateDecoder

• CreateEncoder

  CreateDecoder and CreateEncoder are used to create a decoder/encoder corresponding to a certain data type

  type wrapCodec struct{
      encodeFunc  func(ptr unsafe.Pointer, stream *jsoniter.Stream)
      isEmptyFunc func(ptr unsafe.Pointer) bool
      decodeFunc func(ptr unsafe.Pointer, iter *jsoniter.Iterator)
  }
  
  func (codec *wrapCodec) Encode(ptr unsafe.Pointer, stream *jsoniter.Stream) {
      codec.encodeFunc(ptr, stream)
  }
  
  func (codec *wrapCodec) IsEmpty(ptr unsafe.Pointer) bool {
      if codec.isEmptyFunc == nil {
          return false
      }
  
      return codec.isEmptyFunc(ptr)
  }
  
  func (codec *wrapCodec) Decode(ptr unsafe.Pointer, iter *jsoniter.Iterator) {
      codec.decodeFunc(ptr, iter)
  }
  
  type sampleExtension struct {
      jsoniter.DummyExtension
  }
  
  func (e *sampleExtension) CreateDecoder(typ reflect2.Type) jsoniter.ValDecoder {
      if typ.Kind() == reflect.Int {
          return &wrapCodec{
              decodeFunc:func(ptr unsafe.Pointer, iter *jsoniter.Iterator) {
                  i := iter.ReadInt()
                  *(*int)(ptr) = i - 1000
              },
          }
      }
  
      return nil
  }
  
  func (e *sampleExtension) CreateEncoder(typ reflect2.Type) jsoniter.ValEncoder {
      if typ.Kind() == reflect.Int {
          return &wrapCodec{
              encodeFunc:func(ptr unsafe.Pointer, stream *jsoniter.Stream) {
                  stream.WriteInt(*(*int)(ptr) + 1000)
              },
              isEmptyFunc:nil,
          }
      }
  
      return nil
  }
  
  func extensionTest(){
      i := 20000
      jsoniter.RegisterExtension(&sampleExtension{})
      s, _ := jsoniter.MarshalToString(i)
      fmt.Println(s)
      // Output:
      // 21000
  
      jsoniter.UnmarshalFromString(`30000`, &i)
      fmt.Println(i)
      // Output:
      // 29000
  }

  In the above example, we define wrapCodec who implements ValEncoder and ValDecoder, and then we registered an Extension. In the CreateEncoder function of this Extension, we return a wrapCodec whose Encode function is to add an integer to 1000 before encoding it and flush to the output stream. In the CreateDecoder function of this Extension, we return a wrapCodec whose Decode function is to read an integer from the input stream, then subtract 1000 from it before setting it to the value pointed by ptr

• CreateMapKeyDecoder

• CreateMapKeyEncoder

  The usage of CreateMapKeyDecoder and CreateMapKeyEncoder is similar to the usage of CreateDecoder and CreateEncoder above, except that they are use for the key of type map.

• DecorateDecoder

• DecorateEncoder

  DecorateDecoder and DecorateEncoder can be used to decorate existing ValEncoder and ValEncoder. Considering such an example, on the basis of the examples given in the descriptions of CreateDecoder and CreateEncoder above, we would like to do something more. When we meet a numeric string, we hope that it can also be parsed into integer numbers, and we want to reuse the decoder in the basic example, then we need to use a decorator.

  type decorateExtension struct{
      jsoniter.DummyExtension
  }
  
  type decorateCodec struct{
      originDecoder jsoniter.ValDecoder
  }
  
  func (codec *decorateCodec) Decode(ptr unsafe.Pointer, iter *jsoniter.Iterator) {
      if iter.WhatIsNext() == jsoniter.StringValue {
          str := iter.ReadString()
          if _, err := strconv.Atoi(str); err == nil{
              newIter := iter.Pool().BorrowIterator([]byte(str))
              defer iter.Pool().ReturnIterator(newIter)
              codec.originDecoder.Decode(ptr, newIter)
          }else{
              codec.originDecoder.Decode(ptr, iter)
          }
      } else {
          codec.originDecoder.Decode(ptr, iter)
      }
  }
  
  func (e *decorateExtension) DecorateDecoder(typ reflect2.Type, decoder jsoniter.ValDecoder) jsoniter.ValDecoder{
      if typ.Kind() == reflect.Int {
          return &decorateCodec{decoder}
      }
  
      return nil
  }
  
  func extensionTest(){
      var i int
      jsoniter.RegisterExtension(&sampleExtension{})
      jsoniter.RegisterExtension(&decorateExtension{})
  
      jsoniter.UnmarshalFromString(`30000`, &i)
      fmt.Println(i)
      // Output:
      // 29000
  
      jsoniter.UnmarshalFromString(`"40000"`, &i)
      fmt.Println(i)
      // Output:
      // 39000
  }

  Based on the examples of CreateDecoder and CreateEncoder, we are registering an Extension. This Extension only implements the decorator function, it read numeric string and convert it to integer and it will be subtracted 1000 before being set to the value pointed by ptr

Scope

json-iterator provide two RegisterExtension functions that can be called, one is package level jsoniter.RegisterExtension, and the other is API(see Config chapter) level API.RegisterExtension. Both of them can be used to register extensions, but the scope of extensions registered by the two registration methods is slightly different. The extension registered by jsoniter.RegisterExtension is effective globally， while API.RegisterExtension is only effective for the API interface generated by its corresponding Config.