Out Of Memory: state of the art

Hi All,

Due to popular demand, I will presently describe what we have come up with
investigating OutOfMemory issue (RSRP-18766) that plagues ReSharper since
version 2.0.

As it happens, there can be three different circumstances under which System.OutOfMemory
may be thrown:

1) "True OutOfMemory": ReSharper demands the amount of memory that is greater
that physically available on system (RAM+Pagefile). We believe that on a
reasonable box this never happens in ReSharper 2.5.2. We sometimes eat lots
of memory, but essentialy if you can open your solution in plain VS on given
box without trashing, you shouid be able to do so in VS+ReSharper, although
for small RAMs you will trash a lot. We do work on reducing our memory footprint,
but of course ReSharper features come at a price. You cant expect us to do
more that VS in the same amount of memory. Anyway, I am not aware of ReSharper
encountering 'true OutOfMemory' on any projects.

2) "hResult= E_OUTOFMEMORY". Many COM calls indicate some generic errors
by returning hresult E_OUTOFMEMORY. .NET COM Interop chooses to communicate
this fact to the managed code by throwing OutOfMemory exception.
ReSharper uses Visual Studio COM classes heavily, and, as you may have guessed,
the conditions under which various COM calls fail with E_OUTOFMEMORY are,
to say the least, somewhat underdocumented. What we do in that case is try
to figure out these conditions, not call the offending methods in these cases,
and, as the last resort, just put these call under try/catch(OutOfMemory)
and fail gracefully.
As you may understand, this is an ongoing battle, and we will try to fix
all these calls in 3.0.

3) Large Object Heap troubles. See: http://www.codeproject.com/csharp/Large_Objects____Trouble.asp
Apperently 32bit XP version of .NET 2.0 may fail to allocate big arrays in
some conditions when there are plenty of memory. Worse, Large Obejct Heap
becomes corrupted in that case, and subsequent memory allocation requests
may randomly fail with OutOfMemory exception. ReSharper currently sometimes
tries to allocate big arrays, and we are currenly in the process of replacing
these big arrays with fragmented arrays. I am afraid we will not be able
to backport all of these fixes to 2.5, although we will try to backport the
most critical ones.

These are results of our investigations for RSRP-18766 so far.
Addressing OutOfMemory issues one of our priorities for ReSharper 3.0 release,
and, I believe, we have by now identified the main causes for them and we
hope to fix all of those in 3.0 release.

I do not think there is much need to fly anyone anywhere yet :)

Hope this helps,
Friendly,
Dmitry
--
Dmitry Lomov
Technical Lead/Software Architect
JetBrains, Inc.
"Develop With Pleasure!"


6 comments
Comment actions Permalink

As much as I appreciate you taking the time to sum up the current state of things, and as much as I love R# and want to stick with it, I do have to comment on a couple of statements made in your post:

"As you may understand, this is an ongoing battle, and we will try to fix
all these calls in 3.0."

and

"Addressing OutOfMemory issues one of our priorities for ReSharper 3.0 release,
and, I believe, we have by now identified the main causes for them and we
hope to fix all of those in 3.0 release."

Then I expect you'll either backport your fixes into 2.5 or make 3.0 a free upgrade, because, to be blunt, I have already paid for your product and I most certainly do not remember paying for out of memory exceptions.

0
Comment actions Permalink

Jeremy,
we will certainly backport fixes for the most annoying problems we found.
Do you personally experience lots of OOMs with ReSharper?

Friendly,
Dmitry

As much as I appreciate you taking the time to sum up the current
state of things, and as much as I love R# and want to stick with it, I
do have to comment on a couple of statements made in your post:

"As you may understand, this is an ongoing battle, and we will try to
fix all these calls in 3.0."

and

"Addressing OutOfMemory issues one of our priorities for ReSharper 3.0
release, and, I believe, we have by now identified the main causes for
them and we hope to fix all of those in 3.0 release."

Then I expect you'll either backport your fixes into 2.5 or make 3.0 a
free upgrade, because, to be blunt, I have already paid for your
product and I most certainly do not remember paying for out of memory
exceptions.




--
Dmitry Lomov
Technical Lead/Software Architect
JetBrains, Inc.
"Develop With Pleasure!"


0
Comment actions Permalink

Thankfully, 376 does seem somewhat better in this regard, though I haven't been running it long enough to state conclusively what the degree of improvement is. I just wanted to make sure (and bluntly so ;) ) that one shouldn't be forced to go through a paid upgrade to get this kind of fixes.

As long as you guys are working to get the 2.x line into shape, I'll keep plugging away with R#. I'll just be poking you in the ribs from time to time to make sure. :)

Jeremy

0
Comment actions Permalink

To answer your direct question as to what I have been experiencing (pre 376 evaluation):

R# traditionally chokes and dies at least a couple times per day, having been sluggish well in advance of each death. This is on a 60 project solution with a file count in the approximate realm of 2400. When R# has died like this, VS.NET is not much over 700mb and the machine still has plenty of free memory.

In the context of 376, my gut instinct is that memory usage is down a bit, though I'll have to take some more measurements to be sure, and I haven't had the IDE bog down quite as early or as badly.

Overhead on large source files is still pretty brutal, but frankly I'm willing to put up with that because these very files are in need of serious refactoring that I hope to get rolling during our next project cycle. Still, it never hurts to make R# go faster, especially given that these ugly files are exactly the kind that I want R# to help me tear apart. :)

Jeremy

0
Comment actions Permalink

Jeremy,

Does your 60-project solution involve ASP?

Friendly,
Dmitry

To answer your direct question as to what I have been experiencing
(pre 376 evaluation):

R# traditionally chokes and dies at least a couple times per day,
having been sluggish well in advance of each death. This is on a 60
project solution with a file count in the approximate realm of 2400.
When R# has died like this, VS.NET is not much over 700mb and the
machine still has plenty of free memory.

In the context of 376, my gut instinct is that memory usage is down a
bit, though I'll have to take some more measurements to be sure, and I
haven't had the IDE bog down quite as early or as badly.

Overhead on large source files is still pretty brutal, but frankly I'm
willing to put up with that because these very files are in need of
serious refactoring that I hope to get rolling during our next project
cycle. Still, it never hurts to make R# go faster, especially given
that these ugly files are exactly the kind that I want R# to help me
tear apart. :)

Jeremy

--
Dmitry Lomov
Technical Lead/Software Architect
JetBrains, Inc.
"Develop With Pleasure!"


0
Comment actions Permalink

Does your 60-project solution involve ASP?


I am in no way surprised that you asked that :) and yes, it definitely does. I do my best to avoid opening too many different .aspx or .ascx files at once or even in a given major session during a day, as those seem to be the nastiest of the bunch.

That said, this codebase I've jumped into is in pretty sad shape all around, with the computer slogging its way through multi-thousand line methods]. No, not just multi-thousand line files, multi-thousand line methods. It's hairy stuff all around, so I'm frankly happy that my tools are coping as well as they are. Luckily for me, I got my clean-up effort approved for the next project cycle.

As for my frustration around this issue, I do want to stress that R# is a product that I love, it is just that I want to make sure that the product I've already paid for gets the bugfix attention that it deserves. As long as that continues to be the case, I'll be satisfied. If you can keep it competitive over time (and I think we all know who your killer competition would be) I'll be even happier still. After years of R# it would be very hard to switch, indeed. :)

0

Please sign in to leave a comment.