jackson - JSON Invalid UTF-8 middle byte

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
This error happens when the (Jackson, this case) JSON engine tries to parse some JSON that is not encoded in UTF-8.
How to tell the engine that it should expect something different from UTF-8, such as UTF-16?
HttpHeaders requestHeaders = createSomeHeader();
RestTemplate restTemplate = new RestTemplate();
HttpEntity<?> requestEntity = new HttpEntity<Object>(requestHeaders);
String url = "someurl"
ResponseEntity<MyObject[]> arrayResponseEntity = restTemplate.exchange(url, HttpMethod.GET, requestEntity, MyObject[].class);
error log:
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: Could not read JSON: Invalid UTF-8 middle byte 0x20
at [Source: org.apache.http.conn.EofSensorInputStream@44d397b0; line: 92, column: 42]; nested exception is org.codehaus.jackson.JsonParseException: Invalid UTF-8 middle byte 0x20
at [Source: org.apache.http.conn.EofSensorInputStream@44d397b0; line: 92, column: 42]
at org.springframework.http.converter.json.MappingJacksonHttpMessageConverter.readInternal(MappingJacksonHttpMessageConverter.java:138)
at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:154)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:74)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:622)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:608)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:449)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:404)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:380)
... 4 more
Caused by: org.codehaus.jackson.JsonParseException: Invalid UTF-8 middle byte 0x20
at [Source: org.apache.http.conn.EofSensorInputStream@44d397b0; line: 92, column: 42]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1213)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:375)
at org.codehaus.jackson.impl.Utf8StreamParser._reportInvalidOther(Utf8StreamParser.java:2132)
at org.codehaus.jackson.impl.Utf8StreamParser._reportInvalidOther(Utf8StreamParser.java:2139)
at org.codehaus.jackson.impl.Utf8StreamParser._decodeUtf8_3fast(Utf8StreamParser.java:1962)
I got this exception when in the Java Client Application I was serializing a JSON like this
String json = mapper.writeValueAsString(contentBean);
and on the Server Side I was using Spring Boot as REST Endpoint.
Exception was: 
nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0xaa
My problem was, that I was not setting the correct encoding on the HTTP Client. 
This solved my problem:
updateRequest.setHeader("Content-Type", "application/json;charset=UTF-8");
StringEntity entity= new StringEntity(json, "UTF-8");
updateRequest.setEntity(entity);
Android set content type HttpPost
                This worked for me, Instead of writing out the string, I used. MediaType.APPLICATION_JSON_UTF8_VALUE
– Dot Batch
                Feb 22, 2018 at 23:13
                For anyone getting this error with JIRA, this solution, i.e. adding ;charset=UTF-8, fixed it for me
– user2728841
                Jun 22, 2022 at 18:28
JSON data must be encoded as UTF-8, UTF-16 or UTF-32. The JSON decoder can determine the encoding by examining the first four octets of the byte stream:
       00 00 00 xx  UTF-32BE
       00 xx 00 xx  UTF-16BE
       xx 00 00 00  UTF-32LE
       xx 00 xx 00  UTF-16LE
       xx xx xx xx  UTF-8
It sounds like the server is encoding data in some illegal encoding (ISO-8859-1, windows-1252, etc.)
                And FWIW, this is exactly what Jackson does -- challenge is just that ISO-8859-x does look like UTF-8 in that sense. But since non-unicode encodings are not allowed with Unicode, this is not a problem with valid JSON.
– StaxMan
                Jun 15, 2011 at 20:54
                In my case this was caused by using content.toString().getBytes() for FileObjectStream#write() (where content is a JsonNode) when I created the file - it looks the same, but isnt! The fix is to use new ObjectMapper().writeValueAsBytes(content).  Having realised this, I'm adding it here as a comment in case someone else happens upon this in the same way I did and it helps them
– SteveR
                Nov 5, 2020 at 16:09
On the off chance it may help others I'll share a related anecdote.
I encountered this exact error (Invalid UTF-8 middle byte 0x3f) running a PowerShell script via the PowerShell Integrated Script Environment (ISE). The identical script, executed outside the ISE, works fine. The code uses the Confluence v3 and v5.x REST APIs and this error is logged on the Confluence v5.x server - presumably because the ISE somehow mucks with the request.
  Make sure to start you JVM with -Dfile.encoding=UTF-8. You JVM
  defaults to the operating system charset
This is a JVM argument which could be added, for example, either to JBoss standalone or JBoss running from Eclipse.
In my case, this problem happened isolatelly on only one of my team people's computer. All the others was working without this problem.
I had this problem inconsistently between different platforms, as I got JSON as String from Mapper and did the writing myself. Sometimes it went into file as ansi and other times correctly as UTF8. I switched to 
mapper.writeValue(file, data);
letting Mapper do the file operations, and it started working fine.
POST http://127.0.0.1/bom/create HTTP/1.1
Content-Type: application/json
User-Agent: PostmanRuntime/7.25.0
Accept: */*
Postman-Token: 50ecfbfe-741f-4a2b-a3d3-cdf162ada27f
Host: 127.0.0.1
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 405
  "fwoid": 1,
  "list": [
      "bomIndex": "10001",
      "desc": "带GH 1.25 13pin 公座带针 白色",
      "pn": "084.0001.0036",
      "preUse": 1,
      "type": "追觅 除螨仪-开关PCB组件"
      "bomIndex": "10002",
      "desc": "紫米音箱-商品码标签",
      "pn": "Z.08.013.0051",
      "preUse": 1,
      "type": "E060A0302301"
HTTP/1.1 200 OK
Connection: keep-alive
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
Content-Type: application/json
Date: Mon, 01 Jun 2020 11:23:42 GMT
Content-Length: 40
{"code":"0","message":"BOM保存成功"}
a springboot Controller code as below：
@PostMapping("/bom/create")
@ApiOperation(value = "保存BOM")
@BusinessOperation(module = "BOM",methods = "解析BOM")
public JsonResult save(@RequestBody BOMSaveQuery query)
    return bomService.saveBomList(query);
when i debug on loopback interface,it works ok. while deploy on internet server via bat command, i got an error
ServletInvocableHandlerMethod - Could not resolve parameter [0] in public XXXController.save(com.h2.mes.query.BOMSaveQuery): JSON parse error: Invalid UTF-8 middle byte 0x3f; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 middle byte 0x3f
 at [Source: (PushbackInputStream); line: 9, column: 32] (through reference chain: com.h2.mes.query.BOMSaveQuery["list"]->java.util.ArrayList[0]->com.h2.mes.vo.BOMVO["type"])
2020-06-01 15:37:50.251 MES [XNIO-1 task-13] WARN  o.s.w.s.m.s.DefaultHandlerExceptionResolver - Resolved [org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Invalid UTF-8 middle byte 0x3f; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 middle byte 0x3f
 at [Source: (PushbackInputStream); line: 9, column: 32] (through reference chain: com.h2.mes.query.BOMSaveQuery["list"]->java.util.ArrayList[0]->com.h2.mes.vo.BOMVO["type"])]
2020-06-01 15:37:50.251 MES [XNIO-1 task-13] DEBUG o.s.web.servlet.DispatcherServlet - Completed 400 BAD_REQUEST
2020-06-01 15:37:50.251 MES [XNIO-1 task-13] DEBUG o.s.web.servlet.DispatcherServlet - "ERROR" dispatch for POST "/error", parameters={}
add a jvm arguement works for me.
java -Dfile.encoding=UTF-8
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.