Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I have a serious problem with "JsonConvert.SerializeObject" I need to serialize more than 500,000 dictionary records to make serialize throws the following error; System.OutOfMemoryException. I tried to serialize each key, value pair individually in a foreach but it's locked. Apparently it is an optimization problem, but I do not know where to start, threads to serialize in parts? These functions works fine with few values. My code:

string json = JsonConvert.SerializeObject(DatatableToDictionary(dt), Newtonsoft.Json.Formatting.Indented);
public List<Dictionary<string, object>> DatatableToDictionary(DataTable dt, List<DataColumn> columns)
    return dt.Rows.Cast<DataRow>().Select(
         r => columns.ToDictionary(c => c.ColumnName, c => r[c.ColumnName])).ToList();
                Most likely the total size of the json string to create is too big. Can you show the structure of your dictionary and surrounding types? I was able to easily and quickly write out a million dictionary keys with values to a json file using a stream, a StreamWriter and the JsonTextWriter.
– Lasse V. Karlsen
                Sep 20, 2016 at 15:43
When you're dealing with a large amount of data, you can stream it to a file to avoid having it all in memory at once.
var filePath = @"C:\somewhere.json";
using (var fs = File.Open(filePath, FileMode.CreateNew))
using (var sw = new StreamWriter(fs))
using (var jw = new JsonTextWriter(sw))
    var serializer = new JsonSerializer();
    serializer.Serialize(jw, dictionary);
This will serialize a bit at a time and avoid having a giant string in memory.
                Yes it works fine,I guess then I have to read the file to retrieve the data? because it is a web api, as I'm doing for an MVC application. Is there any way to store information in cache and not in any path?
– gvivetapl
                Sep 20, 2016 at 16:11
                Yes, then you read the file to get data. Reading it is basically the reverse process. The file can basically act as your cache.
– mason
                Sep 20, 2016 at 16:12
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.