[ic] Iterations slow with mv_matchlimit incrementations

[Grant]

>> >> Subject: Re: [ic] Iterations slow with mv_matchlimit incrementations
>> >>
>> >>>>> I have a somewhat complex chunk of IC code about 500 lines long.
>> >>>>> If I
>> >>>>> set the mv_matchlimit of the main loop to a small number, each
>> >>>>> iteration takes less than a second to execute.  However, as I
>> >>>>> increase
>> >>>>> the mv_matchlimit, each iteration takes longer to execute until
>> >> it's
>> >>>>> quite slow.  I've been over my code again and again but I can't
>> >>>>> find a
>> >>>>> cause for this behavior.  The only clue I have is that each
>> >> iteration
>> >>>>> executes much faster when the loop is almost done with all of its
>> >>>>> iterations.
>> >>>>>
>> >>>>> I'm wondering if the problem is "under the hood" of IC.  Does this
>> >>>>> scenario ring a bell for anyone?
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Grant
>> >>>> I'm baffled by this.  I have no idea why increasing mv_matchlimit
>> >>>> would drastically increase the amount of time required for *each*
>> >> loop
>> >>>> iteration.  Please let me know if you have any ideas.
>> >>> Is there any way you can post the relevant piece of code?  Without
>> >>> knowing what's being iterated over, it's hard to offer suggestions.
>> >>> In particular, are there any parts which are perl blocks of any
>> >>> particular flavor (calc, calcn, perl, prefix-exec, etc)?  Are there
>> >>> perhaps multiple nested loop constructs?
>> >>>
>> >>> Regards,
>> >>>
>> >>> David
>> >> Here's another illustration of my problem.  I set up 15 [email] tags
>> >> throughout my code so I get an email whenever certain points are
>> >> reached.  With ml=10 I get maybe 20 or so emails per second.  With
>> >> ml=999999 I get much less than 1 email per second.  My understanding
>> >> is that the first 10 iterations should take the same amount of time in
>> >> either scenario.
>> >>
>> >> Does anyone know why IC would execute a single iteration at a
>> >> drastically slower rate, just because it has more total iterations to
>> >> execute?
>> >>
>> >> My installation is a year or two old.  Does this sound like a problem
>> >> an upgrade could fix?
>> >
>> > e-mail does not seem to be the most useful medium to do performance tests.
>> > Servers can start holding e-mails or other external influences can cause
>> > e-mails to arrive less frequently.
>> >
>> > Why not have 15 timestamps written to a logfile, so you can look at this
>> > data and see if you can find any trends. Is it that the 15 timestamps are
>> > increasing in interval equally?
>> > So it was with
>> > ml=10:   time,  time+I,   time+I+I,  time+I+I+I    (10, 15, 20, 25 etc)
>> > ml=999999:   time,   time+(Ix10),  time+(Ix10+Ix10),  time+(Ix10+Ix10+Ix10)
>> > (10, 60, 110, 160 etc)
>> >
>> > or does ml=999999  show a less regular pattern where it has a spike between
>> > two timings?
>> >
>> > But you can imagine that it is shooting in a black hole here without
>> > actually seeing the code.
>>
>> I agree :-). A possible explanation would be that the memory consumption on
>> larger mv_matchlimit values slows things down (due to problems in the
>> unknown code?).
>>
>
> Two words:
>
>  string copying

Thanks everyone for their help.  I'm working on this and I will of
course report back with my findings.

Mike, could you tell me what you mean?  Is this string copying:

[tmp var1][scratch var2][/tmp]

Maybe this:

[perl]
Scratch->{var1} = $var2;
[/perl]

?

- Grant