| IRC logs at
Set by AlexDaniel on 12 June 2018.
00:02 reportable6 left, reportable6 joined 00:40 MasterDuke left 04:15 japhb left 04:17 japhb joined 06:02 reportable6 left 06:03 reportable6 joined 06:47 squashable6 left 06:48 squashable6 joined 07:51 domidumont joined 08:06 zakharyas joined 08:07 frost-lab joined 09:33 Kaeipi joined 09:34 Merfont left, Kaeipi left 09:35 Kaeipi joined 09:41 Kaeipi left, Kaiepi joined 10:53 zakharyas left 12:01 reportable6 left 12:03 reportable6 joined 13:07 zakharyas joined
lizmat So I think I found a reason why some applications eat much more memory than expected 13:21
on all of the raku logs are read, but the original raw log was *not* kept around
a lot of substr into the raw log are
to shortcut searching, I figured it could be beneficial keeping the original logs in memory 13:22
to my surprise, this did *not* affect memory usage in any notable way
so my conclusion is, is that a substr into a string *is* keeping the original string alive 13:23
is that a correct assumption ? 13:28
jnthn Yes 13:37
lizmat and there is no way to "extract" a string from its "source" ? 13:58
14:14 frost-lab left
jnthn encode/decode 14:18
Will certainly do it, of course at a cost
.NFC.Str too, for a lower cost
lizmat aha,, ok, fair enough...something to keep in mind 14:19
I was originally thinking about instead of keeping substrings, just keeping the raw log and offsets/lengths into that 14:20
now this is clear to me, I realize the system is already doing that for me :-)
timotimo upon garbage collection, we may want to sometimes check if a very long string is being kept around just for a comparatively short substring 14:49
jnthn MoarVM already does expose a way, iirc, to "flatten" a strand string into a single thing, and we could potentially add a named argument to substr like :copy to force it to avoid the optimization in question 14:51
And implement it using that
Or pass it down somehow
timotimo yeah nqp::indexingoptimized is the op 14:55
ideally nobody would have to think about this :D
lizmat well, yeah... actually, now I'm actually keeping the raw log accessible, I don't need it anymore :-) 14:59
timotimo i wonder if there should be access to the rope datastructure's details for users at all 15:10
that sounds like there is now but i'm saying there shouldn't be
i mean, i wonder if it'd be good if users could access the rope's pieces
ugexe is that even possible for non-moarvm backends? 15:12
timotimo no, they would just say "this has one piece, it has 0 repetitions, it starts at 0 and ends at n, and the target string is $self (maybe) 15:13
i guess that could be trouble if you iterate over it?
there'd have to be a signal for "this is a terminal string"? "this is a leaf"? 15:14
lizmat please note I'm fine with how it is now
and it *is* an implementation detail
timotimo mhm
lizmat it's just that it is an extra reason when you're extracting information from a log file, using substr and all 15:15
it's probably better *not* to slurp the whole file first
timotimo even if you slurp it line-by-line, you'd keep each line around if you just hang on to a small substring 15:16
lizmat yeah... there's that
so, if I would run nqp::indexingoptimized on the slurped file contents 15:17
and then discard it
timotimo you'd be expected to call it on everything you substr'd out of it
lizmat ah, ok, so that doesn't make sense then :-)
timotimo it's a little bit of work ;) 15:18
overwrite the substr sub/method to call indexingoptimized on the result, that's how :copy would be implemented
lizmat yeah, got that 15:33
I'm not sure we should, though?
previous discussion: 15:35
16:16 moon-child left 16:21 moon-child joined 17:15 domidumont left 17:53 zakharyas left 18:00 cog_ left 18:02 reportable6 left, reportable6 joined 18:53 cog joined 18:59 lucasb joined 19:09 squashable6 left 19:10 squashable6 joined 19:11 cog left 19:12 cog joined 19:43 MasterDuke joined 20:09 rypervenche left, jpf1 left, rypervenche joined, jpf1 joined 20:38 MasterDuke left 20:44 MasterDuke joined 21:38 MasterDuke left