Memory is great, till you run out

So this morning I had an issue at work where a process kept throwing a System.OutOfMemoryException. We checked out the virtual machine, and seen that the memory being used was about 75-80% of what was available, but that was about it. This had us a bit confused until we looked at the log and seen the exception.

What I found is that in the stack the exception was actually thrown by System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity).  And when I traced this back in the code, I found something a bit concerning (example below)…

string[] headerCell = { "Col1", "Col2" };
var sb = new StringBuilder();
sb.AppendLine(string.Join(",", headerCell));

foreach (var cells in data.Select(row => new string[] { row[0], row[1] }))
        sb.AppendLine(string.Join(",", cells));

    File.WriteAllText(filename, sb.ToString());

At first, this does not seem like a horrible approach, but when you are looping through thousands of rows, the StringBuilder just balloons up and takes up more memory. And of course the ironic part is at the end where we are taking the entire contents and writing it to the file.

To correct this, I implemented a simple StreamWriter object below…

string[] headerCell = { "Col1", "Col2" };
using (var stream = new StreamWriter(filename))
stream.WriteLine(string.Join(",", headerCell));

foreach (var cells in data.Select(row => new string[] { row[0], row[1]}))
                    stream.WriteLine(string.Join(",", cells));

This implementation will loop and write the results directly to the file and will not tie up additional memory while performing the operation.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s