[ic] Iterations slow with mv_matchlimit incrementations

[Grant]

>> >>>>> I have a somewhat complex chunk of IC code about 500 lines long.
>> >>>>> If I
>> >>>>> set the mv_matchlimit of the main loop to a small number, each
>> >>>>> iteration takes less than a second to execute.  However, as I
>> >>>>> increase
>> >>>>> the mv_matchlimit, each iteration takes longer to execute until
>> >> it's
>> >>>>> quite slow.  I've been over my code again and again but I can't
>> >>>>> find a
>> >>>>> cause for this behavior.  The only clue I have is that each
>> >> iteration
>> >>>>> executes much faster when the loop is almost done with all of its
>> >>>>> iterations.
>> >>>>>
>> >>>>> I'm wondering if the problem is "under the hood" of IC.  Does this
>> >>>>> scenario ring a bell for anyone?
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Grant
>> >>>> I'm baffled by this.  I have no idea why increasing mv_matchlimit
>> >>>> would drastically increase the amount of time required for *each*
>> >> loop
>> >>>> iteration.  Please let me know if you have any ideas.
>> >>> Is there any way you can post the relevant piece of code?  Without
>> >>> knowing what's being iterated over, it's hard to offer suggestions.
>> >>> In particular, are there any parts which are perl blocks of any
>> >>> particular flavor (calc, calcn, perl, prefix-exec, etc)?  Are there
>> >>> perhaps multiple nested loop constructs?
>> >>>
>> >>> Regards,
>> >>>
>> >>> David
>> >> Here's another illustration of my problem.  I set up 15 [email] tags
>> >> throughout my code so I get an email whenever certain points are
>> >> reached.  With ml=10 I get maybe 20 or so emails per second.  With
>> >> ml=999999 I get much less than 1 email per second.  My understanding
>> >> is that the first 10 iterations should take the same amount of time in
>> >> either scenario.
>> >>
>> >> Does anyone know why IC would execute a single iteration at a
>> >> drastically slower rate, just because it has more total iterations to
>> >> execute?
>> >>
>> >> My installation is a year or two old.  Does this sound like a problem
>> >> an upgrade could fix?
>> >
>> > e-mail does not seem to be the most useful medium to do performance tests.
>> > Servers can start holding e-mails or other external influences can cause
>> > e-mails to arrive less frequently.
>> >
>> > Why not have 15 timestamps written to a logfile, so you can look at this
>> > data and see if you can find any trends. Is it that the 15 timestamps are
>> > increasing in interval equally?
>> > So it was with
>> > ml=10:   time,  time+I,   time+I+I,  time+I+I+I    (10, 15, 20, 25 etc)
>> > ml=999999:   time,   time+(Ix10),  time+(Ix10+Ix10),  time+(Ix10+Ix10+Ix10)
>> > (10, 60, 110, 160 etc)
>> >
>> > or does ml=999999  show a less regular pattern where it has a spike between
>> > two timings?
>> >
>> > But you can imagine that it is shooting in a black hole here without
>> > actually seeing the code.
>>
>> I agree :-). A possible explanation would be that the memory consumption on
>> larger mv_matchlimit values slows things down (due to problems in the
>> unknown code?).
>>
>
> Two words:
>
>  string copying

I *think* this is what you're talking about:

"Perl is awful for always copying strings around, even if you're never
going to modify them. This wastes CPU and memory. Try to avoid that
whereever reasonably possible. If the string is a function parameter
and the function has a modest length, don't copy the string into a my
variable, access it with $_[0] and document the function well.
Elsewhere, the aliasing feature of for(each) can help. Or just use
references to strings, which are fast to copy. If you somehow ensure
that same strings get stored only once, you can do numerical
comparison for equality."

Can anyone tell me how to put this into action in my ITL coding practices?

- Grant