Dramatically poor performance on very large file

We have a generated class, representing all entities in some financial
scheme,
that has the following characteristics:

size:    2.7 MB
lines:   52023
classes:  124, each containing 1 constructor and a grand total of ...
properties: 6161, each retrieving a value out of a Dictionary , using a
static
read-only property from another class (thus referring to 6161 read-only
members
in another, generated, class) as key.

Just opening the file, takes about:
- 2.5 minutes of 100% CPU before analysis finishes
- 1 more minute to fill the analysis bar with warning marks (6161 warnings)

After removing the warnings (redundant use of keyword base in each
property),
which took about a minute of 5, even with undo temporarily switched off,
opening the file still takes about:
- 40 seconds of 100 CPU%

Some settings:
- File structure window (one of my favorites): closed.
- Highlighting, color identifiers: switched on
- build 253
- version VS2005


Remarks, questions:
- Building the list of warnings seems to be non-linear. I think you can
inprove
  performance dramtically by using the appropriate data structure.
- Mapping the list of warnings to the analysis bar also seems to be rather
  inefficiently.
- Ctrl + 8 does not quickly respond. Would it be possible to disable this at
  forehand for some files (e.g. some sort of settings file remembering the
list of
  files for which it is disabled)?
- Is 40 seconds of 100% CPU acceptabe for a file like this? If not, what can
be
  done about it? (I probably can send you the file personally, but I'm
probably
  not allowed to post it here)


Thanks for any comments,
Erwin


11 comments
Comment actions Permalink

Hello Erwin,

thanks for this information. Could you also please check how long does VS
take to open this file with
ReSharper disabled? Thanks.


Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

We have a generated class, representing all entities in some financial
scheme,
that has the following characteristics:
size:    2.7 MB
lines:   52023
classes:  124, each containing 1 constructor and a grand total of ...
properties: 6161, each retrieving a value out of a Dictionary , using
a
static
read-only property from another class (thus referring to 6161
read-only
members
in another, generated, class) as key.
Just opening the file, takes about:
- 2.5 minutes of 100% CPU before analysis finishes
- 1 more minute to fill the analysis bar with warning marks (6161
warnings)
After removing the warnings (redundant use of keyword base in each
property),
which took about a minute of 5, even with undo temporarily switched
off,
opening the file still takes about:
- 40 seconds of 100 CPU%
Some settings:
- File structure window (one of my favorites): closed.
- Highlighting, color identifiers: switched on
- build 253
- version VS2005
Remarks, questions:
- Building the list of warnings seems to be non-linear. I think you
can
inprove
performance dramtically by using the appropriate data structure.
- Mapping the list of warnings to the analysis bar also seems to be
rather
inefficiently.
- Ctrl + 8 does not quickly respond. Would it be possible to disable
this at
forehand for some files (e.g. some sort of settings file remembering
the
list of
files for which it is disabled)?
- Is 40 seconds of 100% CPU acceptabe for a file like this? If not,
what can
be
done about it? (I probably can send you the file personally, but I'm
probably
not allowed to post it here)
Thanks for any comments,
Erwin



0
Comment actions Permalink

ReSharper disabled: 1 to 2 seconds
Resharper enabled, but "highlight errors and warnings" disabled: 2 seconds

Regards,
Erwin

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in bericht
news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...

Hello Erwin,

>

thanks for this information. Could you also please check how long does VS
take to open this file with
ReSharper disabled? Thanks.

>
>

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>
>> We have a generated class, representing all entities in some financial
>> scheme,
>> that has the following characteristics:
>> size:    2.7 MB
>> lines:   52023
>> classes:  124, each containing 1 constructor and a grand total of ...
>> properties: 6161, each retrieving a value out of a Dictionary , using
>> a
>> static
>> read-only property from another class (thus referring to 6161
>> read-only
>> members
>> in another, generated, class) as key.
>> Just opening the file, takes about:
>> - 2.5 minutes of 100% CPU before analysis finishes
>> - 1 more minute to fill the analysis bar with warning marks (6161
>> warnings)
>> After removing the warnings (redundant use of keyword base in each
>> property),
>> which took about a minute of 5, even with undo temporarily switched
>> off,
>> opening the file still takes about:
>> - 40 seconds of 100 CPU%
>> Some settings:
>> - File structure window (one of my favorites): closed.
>> - Highlighting, color identifiers: switched on
>> - build 253
>> - version VS2005
>> Remarks, questions:
>> - Building the list of warnings seems to be non-linear. I think you
>> can
>> inprove
>> performance dramtically by using the appropriate data structure.
>> - Mapping the list of warnings to the analysis bar also seems to be
>> rather
>> inefficiently.
>> - Ctrl + 8 does not quickly respond. Would it be possible to disable
>> this at
>> forehand for some files (e.g. some sort of settings file remembering
>> the
>> list of
>> files for which it is disabled)?
>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If not,
>> what can
>> be
>> done about it? (I probably can send you the file personally, but I'm
>> probably
>> not allowed to post it here)
>> Thanks for any comments,
>> Erwin
>



0
Comment actions Permalink

Hello Erwin,

can you send us the problematic file or a similar one preserving the negative
performance characteristics of ReSharper on it? Thanks.


Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

ReSharper disabled: 1 to 2 seconds
Resharper enabled, but "highlight errors and warnings" disabled: 2
seconds
Regards,
Erwin
"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...

>> Hello Erwin,
>>
>> thanks for this information. Could you also please check how long
>> does VS
>> take to open this file with
>> ReSharper disabled? Thanks.
>> Regards,
>> Dmitry Shaporenkov
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>>> We have a generated class, representing all entities in some
>>> financial
>>> scheme,
>>> that has the following characteristics:
>>> size:    2.7 MB
>>> lines:   52023
>>> classes:  124, each containing 1 constructor and a grand total of
>>> ...
>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>> using
>>> a
>>> static
>>> read-only property from another class (thus referring to 6161
>>> read-only
>>> members
>>> in another, generated, class) as key.
>>> Just opening the file, takes about:
>>> - 2.5 minutes of 100% CPU before analysis finishes
>>> - 1 more minute to fill the analysis bar with warning marks (6161
>>> warnings)
>>> After removing the warnings (redundant use of keyword base in each
>>> property),
>>> which took about a minute of 5, even with undo temporarily switched
>>> off,
>>> opening the file still takes about:
>>> - 40 seconds of 100 CPU%
>>> Some settings:
>>> - File structure window (one of my favorites): closed.
>>> - Highlighting, color identifiers: switched on
>>> - build 253
>>> - version VS2005
>>> Remarks, questions:
>>> - Building the list of warnings seems to be non-linear. I think you
>>> can
>>> inprove
>>> performance dramtically by using the appropriate data structure.
>>> - Mapping the list of warnings to the analysis bar also seems to be
>>> rather
>>> inefficiently.
>>> - Ctrl + 8 does not quickly respond. Would it be possible to disable
>>> this at
>>> forehand for some files (e.g. some sort of settings file remembering
>>> the
>>> list of
>>> files for which it is disabled)?
>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If not,
>>> what can
>>> be
>>> done about it? (I probably can send you the file personally, but I'm
>>> probably
>>> not allowed to post it here)
>>> Thanks for any comments,
>>> Erwin


0
Comment actions Permalink

Dimitri,

I have mailed you the solution to the address as listed in this message. If
it is a fake mail address please let me know.

Regards,
Erwin

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in bericht
news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...

Hello Erwin,

>

can you send us the problematic file or a similar one preserving the
negative performance characteristics of ReSharper on it? Thanks.

>
>

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>


0
Comment actions Permalink

Hello Erwin,

I've not received your message. My address is dsha(at)jetbrains.com, are
you sure you used it for communication? Thanks.


Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Dimitri,

I have mailed you the solution to the address as listed in this
message. If it is a fake mail address please let me know.

Regards,
Erwin
"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
bericht news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...

>> Hello Erwin,
>>
>> can you send us the problematic file or a similar one preserving the
>> negative performance characteristics of ReSharper on it? Thanks.
>>
>> Regards,
>> Dmitry Shaporenkov
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"


0
Comment actions Permalink

Dimitri,

Thanks for looking into the problem.

I know that the files are extremely large. But as they might be regenerated,
it is not yet possible to split them into smaller chunks (other then to
change the generation proces). I also do not expect you to solve any
performance problem on such extremely large file, but I would like to give
you a few suggestions:

- Some of your algorithms, notably the "building the warning list" and
"mapping the warning list to the analysis bar", do not expose linear
behaviour. Please profile and improve them. Everybody will profit from this.

- Also use large files in your test sets (feel free to add the provided
solution to your test sets, even reducing them by 50 or more %, will still
give you a nice performance test).

- Code generation is becoming more and more part of our daily business.
Think of Micorsoft's DSL tools, MDA tools, etc. So, large files are going to
be part of our daily jobs. Consider to add a feature to disable R# for
specified files at forehand (remember that Ctrl + 8 is not responding
quickly enough).

Hope these suggestions are useful,
anyway, thanks again,

Erwin Derksen

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in bericht
news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...

Hello Erwin,

>

can you send us the problematic file or a similar one preserving the
negative performance characteristics of ReSharper on it? Thanks.

>
>

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>
>> ReSharper disabled: 1 to 2 seconds
>> Resharper enabled, but "highlight errors and warnings" disabled: 2
>> seconds
>> Regards,
>> Erwin
>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>> bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...
>>
>>> Hello Erwin,
>>>
>>> thanks for this information. Could you also please check how long
>>> does VS
>>> take to open this file with
>>> ReSharper disabled? Thanks.
>>> Regards,
>>> Dmitry Shaporenkov
>>> JetBrains, Inc
>>> http://www.jetbrains.com
>>> "Develop with pleasure!"
>>>> We have a generated class, representing all entities in some
>>>> financial
>>>> scheme,
>>>> that has the following characteristics:
>>>> size:    2.7 MB
>>>> lines:   52023
>>>> classes:  124, each containing 1 constructor and a grand total of
>>>> ...
>>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>>> using
>>>> a
>>>> static
>>>> read-only property from another class (thus referring to 6161
>>>> read-only
>>>> members
>>>> in another, generated, class) as key.
>>>> Just opening the file, takes about:
>>>> - 2.5 minutes of 100% CPU before analysis finishes
>>>> - 1 more minute to fill the analysis bar with warning marks (6161
>>>> warnings)
>>>> After removing the warnings (redundant use of keyword base in each
>>>> property),
>>>> which took about a minute of 5, even with undo temporarily switched
>>>> off,
>>>> opening the file still takes about:
>>>> - 40 seconds of 100 CPU%
>>>> Some settings:
>>>> - File structure window (one of my favorites): closed.
>>>> - Highlighting, color identifiers: switched on
>>>> - build 253
>>>> - version VS2005
>>>> Remarks, questions:
>>>> - Building the list of warnings seems to be non-linear. I think you
>>>> can
>>>> inprove
>>>> performance dramtically by using the appropriate data structure.
>>>> - Mapping the list of warnings to the analysis bar also seems to be
>>>> rather
>>>> inefficiently.
>>>> - Ctrl + 8 does not quickly respond. Would it be possible to disable
>>>> this at
>>>> forehand for some files (e.g. some sort of settings file remembering
>>>> the
>>>> list of
>>>> files for which it is disabled)?
>>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If not,
>>>> what can
>>>> be
>>>> done about it? (I probably can send you the file personally, but I'm
>>>> probably
>>>> not allowed to post it here)
>>>> Thanks for any comments,
>>>> Erwin
>



0
Comment actions Permalink

Hello Erwin,

I realize that the files are likely auto-generated and changing the generation
process
is hardly an option. Concerning your suggestions, I don't quite understand
what makes
your think that our algorithms for code analysis are so inefficient. Actually,
they are reasonably
efficient but the problem is not only in algorithms - e.g. such huge files
require large amounts
of memory for data structures related to code analysis. We surely have large
files in our test
harness, although are large files are far from being so huge. However,
automatically disabling code analysis for such large files really seems to
be a plausible thing to
do, and we're considering incorporating it.

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Dimitri,

Thanks for looking into the problem.

I know that the files are extremely large. But as they might be
regenerated, it is not yet possible to split them into smaller chunks
(other then to change the generation proces). I also do not expect you
to solve any performance problem on such extremely large file, but I
would like to give you a few suggestions:

- Some of your algorithms, notably the "building the warning list" and
"mapping the warning list to the analysis bar", do not expose linear
behaviour. Please profile and improve them. Everybody will profit from
this.

- Also use large files in your test sets (feel free to add the
provided solution to your test sets, even reducing them by 50 or more
%, will still give you a nice performance test).

- Code generation is becoming more and more part of our daily
business. Think of Micorsoft's DSL tools, MDA tools, etc. So, large
files are going to be part of our daily jobs. Consider to add a
feature to disable R# for specified files at forehand (remember that
Ctrl + 8 is not responding quickly enough).

Hope these suggestions are useful,
anyway, thanks again,
Erwin Derksen

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
bericht news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...

>> Hello Erwin,
>>
>> can you send us the problematic file or a similar one preserving the
>> negative performance characteristics of ReSharper on it? Thanks.
>>
>> Regards,
>> Dmitry Shaporenkov
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>>> ReSharper disabled: 1 to 2 seconds
>>> Resharper enabled, but "highlight errors and warnings" disabled: 2
>>> seconds
>>> Regards,
>>> Erwin
>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>> bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...
>>>> Hello Erwin,
>>>>
>>>> thanks for this information. Could you also please check how long
>>>> does VS
>>>> take to open this file with
>>>> ReSharper disabled? Thanks.
>>>> Regards,
>>>> Dmitry Shaporenkov
>>>> JetBrains, Inc
>>>> http://www.jetbrains.com
>>>> "Develop with pleasure!"
>>>>> We have a generated class, representing all entities in some
>>>>> financial
>>>>> scheme,
>>>>> that has the following characteristics:
>>>>> size:    2.7 MB
>>>>> lines:   52023
>>>>> classes:  124, each containing 1 constructor and a grand total of
>>>>> ...
>>>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>>>> using
>>>>> a
>>>>> static
>>>>> read-only property from another class (thus referring to 6161
>>>>> read-only
>>>>> members
>>>>> in another, generated, class) as key.
>>>>> Just opening the file, takes about:
>>>>> - 2.5 minutes of 100% CPU before analysis finishes
>>>>> - 1 more minute to fill the analysis bar with warning marks (6161
>>>>> warnings)
>>>>> After removing the warnings (redundant use of keyword base in each
>>>>> property),
>>>>> which took about a minute of 5, even with undo temporarily
>>>>> switched
>>>>> off,
>>>>> opening the file still takes about:
>>>>> - 40 seconds of 100 CPU%
>>>>> Some settings:
>>>>> - File structure window (one of my favorites): closed.
>>>>> - Highlighting, color identifiers: switched on
>>>>> - build 253
>>>>> - version VS2005
>>>>> Remarks, questions:
>>>>> - Building the list of warnings seems to be non-linear. I think
>>>>> you
>>>>> can
>>>>> inprove
>>>>> performance dramtically by using the appropriate data structure.
>>>>> - Mapping the list of warnings to the analysis bar also seems to
>>>>> be
>>>>> rather
>>>>> inefficiently.
>>>>> - Ctrl + 8 does not quickly respond. Would it be possible to
>>>>> disable
>>>>> this at
>>>>> forehand for some files (e.g. some sort of settings file
>>>>> remembering
>>>>> the
>>>>> list of
>>>>> files for which it is disabled)?
>>>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If
>>>>> not,
>>>>> what can
>>>>> be
>>>>> done about it? (I probably can send you the file personally, but
>>>>> I'm
>>>>> probably
>>>>> not allowed to post it here)
>>>>> Thanks for any comments,
>>>>> Erwin


0
Comment actions Permalink

....and it would be definitely useful if you confirm your point of view by
posting a request to our tracker suggesting
to disable ReSharper's code analysis for very large files.

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Hello Erwin,

I realize that the files are likely auto-generated and changing the
generation
process
is hardly an option. Concerning your suggestions, I don't quite
understand
what makes
your think that our algorithms for code analysis are so inefficient.
Actually,
they are reasonably
efficient but the problem is not only in algorithms - e.g. such huge
files
require large amounts
of memory for data structures related to code analysis. We surely have
large
files in our test
harness, although are large files are far from being so huge. However,
automatically disabling code analysis for such large files really
seems to
be a plausible thing to
do, and we're considering incorporating it.
Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>> Dimitri,
>>
>> Thanks for looking into the problem.
>>
>> I know that the files are extremely large. But as they might be
>> regenerated, it is not yet possible to split them into smaller chunks
>> (other then to change the generation proces). I also do not expect
>> you to solve any performance problem on such extremely large file,
>> but I would like to give you a few suggestions:
>>
>> - Some of your algorithms, notably the "building the warning list"
>> and "mapping the warning list to the analysis bar", do not expose
>> linear behaviour. Please profile and improve them. Everybody will
>> profit from this.
>>
>> - Also use large files in your test sets (feel free to add the
>> provided solution to your test sets, even reducing them by 50 or more
>> %, will still give you a nice performance test).
>>
>> - Code generation is becoming more and more part of our daily
>> business. Think of Micorsoft's DSL tools, MDA tools, etc. So, large
>> files are going to be part of our daily jobs. Consider to add a
>> feature to disable R# for specified files at forehand (remember that
>> Ctrl + 8 is not responding quickly enough).
>>
>> Hope these suggestions are useful,
>> anyway, thanks again,
>> Erwin Derksen
>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>> bericht news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...
>>
>>> Hello Erwin,
>>>
>>> can you send us the problematic file or a similar one preserving the
>>> negative performance characteristics of ReSharper on it? Thanks.
>>>
>>> Regards,
>>> Dmitry Shaporenkov
>>> JetBrains, Inc
>>> http://www.jetbrains.com
>>> "Develop with pleasure!"
>>>> ReSharper disabled: 1 to 2 seconds
>>>> Resharper enabled, but "highlight errors and warnings" disabled: 2
>>>> seconds
>>>> Regards,
>>>> Erwin
>>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>>> bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...
>>>>> Hello Erwin,
>>>>>
>>>>> thanks for this information. Could you also please check how long
>>>>> does VS
>>>>> take to open this file with
>>>>> ReSharper disabled? Thanks.
>>>>> Regards,
>>>>> Dmitry Shaporenkov
>>>>> JetBrains, Inc
>>>>> http://www.jetbrains.com
>>>>> "Develop with pleasure!"
>>>>>> We have a generated class, representing all entities in some
>>>>>> financial
>>>>>> scheme,
>>>>>> that has the following characteristics:
>>>>>> size:    2.7 MB
>>>>>> lines:   52023
>>>>>> classes:  124, each containing 1 constructor and a grand total of
>>>>>> ...
>>>>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>>>>> using
>>>>>> a
>>>>>> static
>>>>>> read-only property from another class (thus referring to 6161
>>>>>> read-only
>>>>>> members
>>>>>> in another, generated, class) as key.
>>>>>> Just opening the file, takes about:
>>>>>> - 2.5 minutes of 100% CPU before analysis finishes
>>>>>> - 1 more minute to fill the analysis bar with warning marks (6161
>>>>>> warnings)
>>>>>> After removing the warnings (redundant use of keyword base in
>>>>>> each
>>>>>> property),
>>>>>> which took about a minute of 5, even with undo temporarily
>>>>>> switched
>>>>>> off,
>>>>>> opening the file still takes about:
>>>>>> - 40 seconds of 100 CPU%
>>>>>> Some settings:
>>>>>> - File structure window (one of my favorites): closed.
>>>>>> - Highlighting, color identifiers: switched on
>>>>>> - build 253
>>>>>> - version VS2005
>>>>>> Remarks, questions:
>>>>>> - Building the list of warnings seems to be non-linear. I think
>>>>>> you
>>>>>> can
>>>>>> inprove
>>>>>> performance dramtically by using the appropriate data structure.
>>>>>> - Mapping the list of warnings to the analysis bar also seems to
>>>>>> be
>>>>>> rather
>>>>>> inefficiently.
>>>>>> - Ctrl + 8 does not quickly respond. Would it be possible to
>>>>>> disable
>>>>>> this at
>>>>>> forehand for some files (e.g. some sort of settings file
>>>>>> remembering
>>>>>> the
>>>>>> list of
>>>>>> files for which it is disabled)?
>>>>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If
>>>>>> not,
>>>>>> what can
>>>>>> be
>>>>>> done about it? (I probably can send you the file personally, but
>>>>>> I'm
>>>>>> probably
>>>>>> not allowed to post it here)
>>>>>> Thanks for any comments,
>>>>>> Erwin



0
Comment actions Permalink

Hi,

What about splitting the analysis into two parts, that can be
enabled/disabled by the user. Checking for errors and checking for warnings.
Most of the time when working in sourcecode, I am interested in realtime
error indicators, but could manage without realtime warnings.

/peter

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> wrote in message
news:c8a894d91a60cf8c86f11e1b87826@news.intellij.net...

...and it would be definitely useful if you confirm your point of view by
posting a request to our tracker suggesting
to disable ReSharper's code analysis for very large files.

>

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>
>> Hello Erwin,
>>
>> I realize that the files are likely auto-generated and changing the
>> generation
>> process
>> is hardly an option. Concerning your suggestions, I don't quite
>> understand
>> what makes
>> your think that our algorithms for code analysis are so inefficient.
>> Actually,
>> they are reasonably
>> efficient but the problem is not only in algorithms - e.g. such huge
>> files
>> require large amounts
>> of memory for data structures related to code analysis. We surely have
>> large
>> files in our test
>> harness, although are large files are far from being so huge. However,
>> automatically disabling code analysis for such large files really
>> seems to
>> be a plausible thing to
>> do, and we're considering incorporating it.
>> Regards,
>> Dmitry Shaporenkov
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>>> Dimitri,
>>>
>>> Thanks for looking into the problem.
>>>
>>> I know that the files are extremely large. But as they might be
>>> regenerated, it is not yet possible to split them into smaller chunks
>>> (other then to change the generation proces). I also do not expect
>>> you to solve any performance problem on such extremely large file,
>>> but I would like to give you a few suggestions:
>>>
>>> - Some of your algorithms, notably the "building the warning list"
>>> and "mapping the warning list to the analysis bar", do not expose
>>> linear behaviour. Please profile and improve them. Everybody will
>>> profit from this.
>>>
>>> - Also use large files in your test sets (feel free to add the
>>> provided solution to your test sets, even reducing them by 50 or more
>>> %, will still give you a nice performance test).
>>>
>>> - Code generation is becoming more and more part of our daily
>>> business. Think of Micorsoft's DSL tools, MDA tools, etc. So, large
>>> files are going to be part of our daily jobs. Consider to add a
>>> feature to disable R# for specified files at forehand (remember that
>>> Ctrl + 8 is not responding quickly enough).
>>>
>>> Hope these suggestions are useful,
>>> anyway, thanks again,
>>> Erwin Derksen
>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>> bericht news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...
>>>
>>>> Hello Erwin,
>>>>
>>>> can you send us the problematic file or a similar one preserving the
>>>> negative performance characteristics of ReSharper on it? Thanks.
>>>>
>>>> Regards,
>>>> Dmitry Shaporenkov
>>>> JetBrains, Inc
>>>> http://www.jetbrains.com
>>>> "Develop with pleasure!"
>>>>> ReSharper disabled: 1 to 2 seconds
>>>>> Resharper enabled, but "highlight errors and warnings" disabled: 2
>>>>> seconds
>>>>> Regards,
>>>>> Erwin
>>>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>>>> bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...
>>>>>> Hello Erwin,
>>>>>>
>>>>>> thanks for this information. Could you also please check how long
>>>>>> does VS
>>>>>> take to open this file with
>>>>>> ReSharper disabled? Thanks.
>>>>>> Regards,
>>>>>> Dmitry Shaporenkov
>>>>>> JetBrains, Inc
>>>>>> http://www.jetbrains.com
>>>>>> "Develop with pleasure!"
>>>>>>> We have a generated class, representing all entities in some
>>>>>>> financial
>>>>>>> scheme,
>>>>>>> that has the following characteristics:
>>>>>>> size:    2.7 MB
>>>>>>> lines:   52023
>>>>>>> classes:  124, each containing 1 constructor and a grand total of
>>>>>>> ...
>>>>>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>>>>>> using
>>>>>>> a
>>>>>>> static
>>>>>>> read-only property from another class (thus referring to 6161
>>>>>>> read-only
>>>>>>> members
>>>>>>> in another, generated, class) as key.
>>>>>>> Just opening the file, takes about:
>>>>>>> - 2.5 minutes of 100% CPU before analysis finishes
>>>>>>> - 1 more minute to fill the analysis bar with warning marks (6161
>>>>>>> warnings)
>>>>>>> After removing the warnings (redundant use of keyword base in
>>>>>>> each
>>>>>>> property),
>>>>>>> which took about a minute of 5, even with undo temporarily
>>>>>>> switched
>>>>>>> off,
>>>>>>> opening the file still takes about:
>>>>>>> - 40 seconds of 100 CPU%
>>>>>>> Some settings:
>>>>>>> - File structure window (one of my favorites): closed.
>>>>>>> - Highlighting, color identifiers: switched on
>>>>>>> - build 253
>>>>>>> - version VS2005
>>>>>>> Remarks, questions:
>>>>>>> - Building the list of warnings seems to be non-linear. I think
>>>>>>> you
>>>>>>> can
>>>>>>> inprove
>>>>>>> performance dramtically by using the appropriate data structure.
>>>>>>> - Mapping the list of warnings to the analysis bar also seems to
>>>>>>> be
>>>>>>> rather
>>>>>>> inefficiently.
>>>>>>> - Ctrl + 8 does not quickly respond. Would it be possible to
>>>>>>> disable
>>>>>>> this at
>>>>>>> forehand for some files (e.g. some sort of settings file
>>>>>>> remembering
>>>>>>> the
>>>>>>> list of
>>>>>>> files for which it is disabled)?
>>>>>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If
>>>>>>> not,
>>>>>>> what can
>>>>>>> be
>>>>>>> done about it? (I probably can send you the file personally, but
>>>>>>> I'm
>>>>>>> probably
>>>>>>> not allowed to post it here)
>>>>>>> Thanks for any comments,
>>>>>>> Erwin
>
>



0
Comment actions Permalink

Hello Peter,

I've answered you via e-mail. Basically, ReSharper code analysis has a certain
architecture and splitting it into
separate passes for finding errors and warnings doesn't fit in it. However,
the principle 'most important errors (like unresolved symbols) are found
first' is more or less
sustained.

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Hi,

What about splitting the analysis into two parts, that can be
enabled/disabled by the user. Checking for errors and checking for
warnings. Most of the time when working in sourcecode, I am interested
in realtime error indicators, but could manage without realtime
warnings.

/peter

"Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> wrote in message
news:c8a894d91a60cf8c86f11e1b87826@news.intellij.net...

>> ...and it would be definitely useful if you confirm your point of
>> view by
>> posting a request to our tracker suggesting
>> to disable ReSharper's code analysis for very large files.
>> Regards,
>> Dmitry Shaporenkov
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>>> Hello Erwin,
>>>
>>> I realize that the files are likely auto-generated and changing the
>>> generation
>>> process
>>> is hardly an option. Concerning your suggestions, I don't quite
>>> understand
>>> what makes
>>> your think that our algorithms for code analysis are so inefficient.
>>> Actually,
>>> they are reasonably
>>> efficient but the problem is not only in algorithms - e.g. such huge
>>> files
>>> require large amounts
>>> of memory for data structures related to code analysis. We surely
>>> have
>>> large
>>> files in our test
>>> harness, although are large files are far from being so huge.
>>> However,
>>> automatically disabling code analysis for such large files really
>>> seems to
>>> be a plausible thing to
>>> do, and we're considering incorporating it.
>>> Regards,
>>> Dmitry Shaporenkov
>>> JetBrains, Inc
>>> http://www.jetbrains.com
>>> "Develop with pleasure!"
>>>> Dimitri,
>>>>
>>>> Thanks for looking into the problem.
>>>>
>>>> I know that the files are extremely large. But as they might be
>>>> regenerated, it is not yet possible to split them into smaller
>>>> chunks (other then to change the generation proces). I also do not
>>>> expect you to solve any performance problem on such extremely large
>>>> file, but I would like to give you a few suggestions:
>>>>
>>>> - Some of your algorithms, notably the "building the warning list"
>>>> and "mapping the warning list to the analysis bar", do not expose
>>>> linear behaviour. Please profile and improve them. Everybody will
>>>> profit from this.
>>>>
>>>> - Also use large files in your test sets (feel free to add the
>>>> provided solution to your test sets, even reducing them by 50 or
>>>> more %, will still give you a nice performance test).
>>>>
>>>> - Code generation is becoming more and more part of our daily
>>>> business. Think of Micorsoft's DSL tools, MDA tools, etc. So, large
>>>> files are going to be part of our daily jobs. Consider to add a
>>>> feature to disable R# for specified files at forehand (remember
>>>> that Ctrl + 8 is not responding quickly enough).
>>>>
>>>> Hope these suggestions are useful,
>>>> anyway, thanks again,
>>>> Erwin Derksen
>>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>>> bericht news:c8a894d91a189a8c86b71ce0e547a@news.intellij.net...
>>>>> Hello Erwin,
>>>>>
>>>>> can you send us the problematic file or a similar one preserving
>>>>> the negative performance characteristics of ReSharper on it?
>>>>> Thanks.
>>>>>
>>>>> Regards,
>>>>> Dmitry Shaporenkov
>>>>> JetBrains, Inc
>>>>> http://www.jetbrains.com
>>>>> "Develop with pleasure!"
>>>>>> ReSharper disabled: 1 to 2 seconds
>>>>>> Resharper enabled, but "highlight errors and warnings" disabled:
>>>>>> 2
>>>>>> seconds
>>>>>> Regards,
>>>>>> Erwin
>>>>>> "Dmitry Shaporenkov (JetBrains)" <dsha@jetbrains.com> schreef in
>>>>>> bericht news:c8a894d91a0ddf8c86afbfde16dfb@news.intellij.net...
>>>>>>> Hello Erwin,
>>>>>>>
>>>>>>> thanks for this information. Could you also please check how
>>>>>>> long
>>>>>>> does VS
>>>>>>> take to open this file with
>>>>>>> ReSharper disabled? Thanks.
>>>>>>> Regards,
>>>>>>> Dmitry Shaporenkov
>>>>>>> JetBrains, Inc
>>>>>>> http://www.jetbrains.com
>>>>>>> "Develop with pleasure!"
>>>>>>>> We have a generated class, representing all entities in some
>>>>>>>> financial
>>>>>>>> scheme,
>>>>>>>> that has the following characteristics:
>>>>>>>> size:    2.7 MB
>>>>>>>> lines:   52023
>>>>>>>> classes:  124, each containing 1 constructor and a grand total
>>>>>>>> of
>>>>>>>> ...
>>>>>>>> properties: 6161, each retrieving a value out of a Dictionary ,
>>>>>>>> using
>>>>>>>> a
>>>>>>>> static
>>>>>>>> read-only property from another class (thus referring to 6161
>>>>>>>> read-only
>>>>>>>> members
>>>>>>>> in another, generated, class) as key.
>>>>>>>> Just opening the file, takes about:
>>>>>>>> - 2.5 minutes of 100% CPU before analysis finishes
>>>>>>>> - 1 more minute to fill the analysis bar with warning marks
>>>>>>>> (6161
>>>>>>>> warnings)
>>>>>>>> After removing the warnings (redundant use of keyword base in
>>>>>>>> each
>>>>>>>> property),
>>>>>>>> which took about a minute of 5, even with undo temporarily
>>>>>>>> switched
>>>>>>>> off,
>>>>>>>> opening the file still takes about:
>>>>>>>> - 40 seconds of 100 CPU%
>>>>>>>> Some settings:
>>>>>>>> - File structure window (one of my favorites): closed.
>>>>>>>> - Highlighting, color identifiers: switched on
>>>>>>>> - build 253
>>>>>>>> - version VS2005
>>>>>>>> Remarks, questions:
>>>>>>>> - Building the list of warnings seems to be non-linear. I think
>>>>>>>> you
>>>>>>>> can
>>>>>>>> inprove
>>>>>>>> performance dramtically by using the appropriate data
>>>>>>>> structure.
>>>>>>>> - Mapping the list of warnings to the analysis bar also seems
>>>>>>>> to
>>>>>>>> be
>>>>>>>> rather
>>>>>>>> inefficiently.
>>>>>>>> - Ctrl + 8 does not quickly respond. Would it be possible to
>>>>>>>> disable
>>>>>>>> this at
>>>>>>>> forehand for some files (e.g. some sort of settings file
>>>>>>>> remembering
>>>>>>>> the
>>>>>>>> list of
>>>>>>>> files for which it is disabled)?
>>>>>>>> - Is 40 seconds of 100% CPU acceptabe for a file like this? If
>>>>>>>> not,
>>>>>>>> what can
>>>>>>>> be
>>>>>>>> done about it? (I probably can send you the file personally,
>>>>>>>> but
>>>>>>>> I'm
>>>>>>>> probably
>>>>>>>> not allowed to post it here)
>>>>>>>> Thanks for any comments,
>>>>>>>> Erwin


0
Comment actions Permalink

...and it would be definitely useful if you confirm your point of view by
posting a request to our tracker suggesting
to disable ReSharper's code analysis for very large files.

>

#27094

what makes
your think that our algorithms for code analysis are so inefficient.


Well, opening the file takes:
- 2.5 minutes of 100% CPU before analysis finishes
- 1 more minute to fill the analysis bar with warning marks (6161 warnings)

After removing the warnings (6161 redundant usages of keyword base)
- 40 seconds of 100 CPU%

This is a difference of 110 seconds, while the file itself only lost 5*6161
characters. That cannot be the cause of such a difference for analysis. I
think, but forgive if I'm wrong, that somewhere in filling the data
structure with warnings some non-linear behaviour occurs. This can be in the
.Net collections themselves. It is e.g. known that the DataTable of .Net 1.1
does not scale well. But it can also be the choice of the data structure.

Regards,
Erwin


0

Please sign in to leave a comment.