[ic] Iterations slow with mv_matchlimit incrementations

[Peter]

On 07/05/2009 07:52 AM, Gert van der Spoel wrote:
> We were looking at the overall speed/performance of loops a bit today
> (Peter, Phil, myself) and Peter gave some valuable tips, which I can also be
> found in the following pages where you might find some other useful
> information as well:
> 
> http://www.icdevgroup.org/docs/optimization.html
> http://www.interchange.rtfm.info/icdocs/Interchange_tag_introduction.html#In
> terchange_tag_parsing_order
> http://www.interchange.rtfm.info/icdocs/Interchange_Perl_objects.html  and
> then specifically:
> http://www.interchange.rtfm.info/icdocs/Interchange_Perl_objects.html#Sql

Also something to keep in mind is the parsing order of tags and looping
tags (refer to the 2nd link above).  If you do something like this:
[loop blah blah]
[time]%T[/time]
[/loop]


...then you may not realize that the [time] tag is not actually being
run at all until the loop is completely finished processing.  Therefore
the times you get will not be indicative of the actual time that each
loop is being run, something like this is more likely to be what you want:
[loop-calc]$Tag->time({body => '%T'})[/loop-calc]

The above will cause teh time tag to run at loop time instead of after.

Also to keep in mind is that the way that loop actually works is that
all text including non-looping tags get piled up into one gigantic
string, then that string is passed to the parser in one shot and all
tags have to then be parsed.  If you have several non-looping tags
inside your loop and you are processing upwards of 100 or so loop
iterations, (and it would seem you are trying to process on an order of
hundreds of thousands, or at least tens of thousands), then IC will end
up putting together such a long string that it will take up significant
memory (as you have observed).  After putting the string together IC has
to them parse the non-looping tags through the use of regular
expressions.  This will slow things down enough, but if you're running
perl 5.10.0 and the string happens to get flagged as utf8 along the way
somewhere then there is a known bug in 5.10.0 that seriously slows down
the regexp parsing in many situations(someone will give a link to it,
I'm not gonna look it up right now).  This would certainly explain the
slow downs you are experiencing.

The first link above gives some good help in avoiding this problem.
What you should consider is try to avoid non-looping tags as much as
possible, and if you can use a [perl] block and write your loop in perl
instead of using the loop tag.  The third link above can give some
useful perl objects to use in your perl blocks.  Also if you want to use
a non-looping tag inside a [loop] block you can usually do so with the
[loop-calc] method shown above for the [time] tag.

The idea with the above optimizations is to try to do your processing
during the loop (good) instead of piling up a massive string and trying
to process it all later (bad).


Peter