[SpringBoot/SpringMVC]从Webapp下载一个大文件出现java.lang.OutOfMemoryError: GC overhead limit exceeded怎么办？
mb5fdb1021b5992
[SpringBoot/SpringMVC]从Webapp下载一个大文件出现java.lang.OutOfMemoryError: GC overhead limit exceeded怎么办？
制作一个Webapp，让其中一个网页提供下载链接，以使用户能下载本地文件或是临时生成的文件，这些都不是难事，网上也有很多既存的解决方案。
但是，这个问题难点在，但生成文件过大时，产生java.lang.OutOfMemoryError异常怎么办？有人提出修改JVM内存参数，如-Xms<min>m -Xmx<max>m方式，但终究是治标不治本的方法，如果下载数据量又超过设定的上限呢？
其实问题的本质是：在为大数据生成准备过程中，大量对象产生了来不及释放，因为还需要在接下来的步骤中使用，所以驻留在内存中，内存越积越多，终究导致java.lang.OutOfMemoryError异常。举个例子来说，有个emp表，存储员工的id姓名年龄等，当只有千百条时，取出结果集转成链表再写入csv文件自然没什么问题，但如果数据越来越多，还是结果集放链表里又来不及释放，总有内存不够的时候。
2019-10-25 14:43:05.518 ERROR 40016 --- [nio-8080-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded] with root cause
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.Integer.toString(Unknown Source) ~[na:1.8.0_201]
at java.lang.String.valueOf(Unknown Source) ~[na:1.8.0_201]
at com.example.demo.CsvUtil.exportBigCsv(CsvUtil.java:78) ~[classes/:na]
at com.example.demo.SpringBootWeb1Application.downloadBigCsvFile(SpringBootWeb1Application.java:141) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_201]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
要解决这个问题，就得耐下心来，将整体读整体写改成分批读分批写（分批方案和分页方案类似，诸位可自行寻找合适自己DB的DB dialect），最后再下载。
示例页面：
<!DOCTYPE html>
<html lang="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>happy go lucky</title>
</head>
<h2>文件下载测试</h2>
<li><a href="/dldLocalFile">下载本地文件dld.rar</a></li>
<li><a href="/dldGeneratedFile/100">下载生成百行CSV文件</a></li>
<li><a href="/dldGeneratedFile/100000000">下载生成亿行CSV文件</a></li>
<li><a href="/dldGeneratedFile2/10000000">下载生成千万行CSV文件 可能 出现java.lang.OutOfMemoryError: GC overhead limit exceeded</a></li>
<li><a href="/dldGeneratedFile3/10000000">下载生成千万行CSV文件 不会可能 出现java.lang.OutOfMemoryError: GC overhead limit exceeded</a></li>
</body>
</html>
<script type="text/javascript">
// 脚本
</script>
package com.example.demo;
import java.io.File;
import java.io.FileInputStream;
import java.net.URLDecoder;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.log4j.Logger;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
@Controller
@SpringBootApplication
public class WebFileDownloadApplication {
private static Logger logger = Logger.getLogger(WebFileDownloadApplication.class);
public static void main(String[] args) {
SpringApplication.run(WebFileDownloadApplication.class, args);
@RequestMapping("/")
public String index() {
logger.info("进入index页");
return "index.html";
@RequestMapping("/dldLocalFile")
public void downloadLocalFile(HttpServletResponse res, HttpServletRequest req) throws Exception {
logger.info("下载本地文件");
String localFilename = "dld.rar";
String localFilepath = getClass().getResource("/static/" + localFilename).getPath();
res.setContentType("multipart/form-data");
res.setCharacterEncoding("UTF-8");
res.setContentType("text/html");
String userAgent = req.getHeader("User-Agent");
if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
// IE Core
localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
} else {
// Non-IE Core
localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);
localFilepath = URLDecoder.decode(localFilepath, "UTF-8");
FileInputStream instream = new FileInputStream(localFilepath);
ServletOutputStream outstream = res.getOutputStream();
int b = 0;
byte[] buffer = new byte[1024];
while ((b = instream.read(buffer)) != -1) {
outstream.write(buffer, 0, b);
instream.close();
if (outstream != null) {
outstream.flush();
outstream.close();
@RequestMapping("/dldGeneratedFile/{count}")
public void downloadGeneratedCsvFile(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
logger.info("Start downloadGeneratedCsvFile");
SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
Date time = new Date();
String tStamp = dfs.format(time);
String localFilename = tStamp+".csv";
String path=req.getSession().getServletContext().getRealPath("/");
String localFilepath = path+localFilename;
logger.info("准备生成的本地路径文件名="+localFilepath);
res.setContentType("multipart/form-data");
res.setCharacterEncoding("UTF-8");
res.setContentType("text/html");
String userAgent = req.getHeader("User-Agent");
if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
// IE Core
localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
} else {
// Non-IE Core
localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);
localFilepath = URLDecoder.decode(localFilepath, "UTF-8");
List<String> dataList=new ArrayList<String>();
for(int i=0;i<100;i++) {
dataList.add(String.valueOf(i));
File file=new File(localFilepath);
CsvUtil.generateCsv(file,Integer.parseInt(count));
logger.info("已经生成文件:"+localFilepath);
FileInputStream instream = new FileInputStream(localFilepath);
ServletOutputStream outstream = res.getOutputStream();
int b = 0;
byte[] buffer = new byte[1024];
while ((b = instream.read(buffer)) != -1) {
outstream.write(buffer, 0, b);
instream.close();
if (outstream != null) {
outstream.flush();
outstream.close();
boolean isDeleted=file.delete();
if(isDeleted) {
logger.info("已经删除文件："+localFilepath);
@RequestMapping("/dldGeneratedFile2/{count}")
public void downloadGeneratedCsvFile2(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
logger.info("Start downloadGeneratedCsvFile2");
SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
Date time = new Date();
String tStamp = dfs.format(time);
String localFilename = tStamp+".csv";
String path=req.getSession().getServletContext().getRealPath("/");
String localFilepath = path+localFilename;
logger.info("准备生成的本地路径文件名="+localFilepath);
res.setContentType("multipart/form-data");
res.setCharacterEncoding("UTF-8");
res.setContentType("text/html");
String userAgent = req.getHeader("User-Agent");
if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
// IE Core
localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
} else {
// Non-IE Core
localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);
localFilepath = URLDecoder.decode(localFilepath, "UTF-8");
List<String> dataList=new ArrayList<String>();
for(int i=0;i<100;i++) {
dataList.add(String.valueOf(i));
File file=new File(localFilepath);
CsvUtil.generateCsv2(file,Integer.parseInt(count));
logger.info("已经生成文件:"+localFilepath);
FileInputStream instream = new FileInputStream(localFilepath);
ServletOutputStream outstream = res.getOutputStream();
int b = 0;
byte[] buffer = new byte[1024];
while ((b = instream.read(buffer)) != -1) {
outstream.write(buffer, 0, b);
instream.close();
if (outstream != null) {
outstream.flush();
outstream.close();
boolean isDeleted=file.delete();
if(isDeleted) {
logger.info("已经删除文件："+localFilepath);
@RequestMapping("/dldGeneratedFile3/{count}")
public void downloadGeneratedCsvFile3(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
logger.info("Start downloadGeneratedCsvFile3");
SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
Date time = new Date();
String tStamp = dfs.format(time);
String localFilename = tStamp+".csv";
String path=req.getSession().getServletContext().getRealPath("/");
String localFilepath = path+localFilename;
logger.info("准备生成的本地路径文件名="+localFilepath);
res.setContentType("multipart/form-data");
res.setCharacterEncoding("UTF-8");
res.setContentType("text/html");
String userAgent = req.getHeader("User-Agent");
if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
// IE Core
localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
} else {
// Non-IE Core
localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);
localFilepath = URLDecoder.decode(localFilepath, "UTF-8");
List<String> dataList=new ArrayList<String>();
for(int i=0;i<100;i++) {
dataList.add(String.valueOf(i));
File file=new File(localFilepath);
CsvUtil.generateCsv3(file,Integer.parseInt(count));
logger.info("已经生成文件:"+localFilepath);
FileInputStream instream = new FileInputStream(localFilepath);
ServletOutputStream outstream = res.getOutputStream();
int b = 0;
byte[] buffer = new byte[1024];
while ((b = instream.read(buffer)) != -1) {
outstream.write(buffer, 0, b);
instream.close();
if (outstream != null) {
outstream.flush();
outstream.close();
boolean isDeleted=file.delete();
if(isDeleted) {
logger.info("已经删除文件："+localFilepath);
数据生成类：
package com.example.demo;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.log4j.Logger;
public class CsvUtil {
private static Logger logger = Logger.getLogger(CsvUtil.class);
// 这种方式不会内存溢出，因为创建的数组对象用于写完文件后在一个循环结束就释放了
public static boolean generateCsv(File file,int rowCount) {
try {
FileWriter fileWriter =new FileWriter(file, true);
for(int j=0;j<rowCount;j++) {
int lineNo=j;
String[] arr=new String[20];
for(int k=0;k<arr.length;k++) {
arr[k]=String.valueOf(k);
String info=String.valueOf(lineNo)+","+String.join(",", arr)+System.getProperty("line.separator");
fileWriter.write(info);
fileWriter.flush();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
return true;
// 这种方式因为生成一个大list，容易造成内存驻留过多而溢出
public static boolean generateCsv2(File file,int rowCount) {
try {
// 先模拟访问数据库创建集合，当数据量大时会出现outofmemory异常
List<String[]> list=new ArrayList<String[]>();
for(int i=0;i<rowCount;i++) {
String[] arr=new String[20];
for(int k=0;k<arr.length;k++) {
arr[k]=String.valueOf(k);
list.add(arr);
// 再将集合写文件
FileWriter fileWriter =new FileWriter(file, true);
int index=0;
for(String[] arr:list) {
index++;
String info=String.valueOf(index)+","+String.join(",", arr)+System.getProperty("line.separator");
fileWriter.write(info);
fileWriter.flush();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
return true;
// 这种方式改成分批写入文件方式，以规避内存溢出
public static boolean generateCsv3(File file,int rowCount) {
logger.info("generateCsv3");
try {
final int LinesOnceTime=10000;
int count=rowCount/LinesOnceTime;
for(int i=0;i<count;i++) {
// 再将集合写文件
FileWriter fileWriter =new FileWriter(file, true);
List<String[]> list=new ArrayList<String[]>();
// 这回list被限制在了LinesOnceTime件,这一步代表着从数据库分批取，oracle有townum支持，mysql有limit支持
for(int j=0;j<LinesOnceTime;j++) {
String[] arr=new String[20];
for(int k=0;k<arr.length;k++) {
arr[k]=String.valueOf(k);
list.add(arr);
int index=0;
for(String[] arr:list) {
index++;
int lineNo=i*LinesOnceTime+index;
String info=String.valueOf(lineNo)+","+String.join(",", arr)+System.getProperty("line.separator");
fileWriter.write(info);
fileWriter.flush();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
return true;
--END-- 2019年10月26日14:49:02
What means the error-message 'java.lang.OutOfMemoryError: GC overhead limit exceeded' in Java?
一、异常如下：Exception in thread "main" java.lang.OutOfMemoryError: G
