samcv | well, as a pointer to some malloced grapheme iterator i suppose. so we don't have to refind our place every time. since perl 6 or nqp let's say it requests graphemes one after each other, then it colud be faster to save the iterator | 00:10 | |
MasterDuke | i'm just spouting stuff off the top of my head here, but if MVMStrings have to be immutable, what about creating a small pool of grapheme iterators? maybe accessed by the MVMStringBody's cached_hash_code? | 00:25 | |
samcv | well i'm going to work on getting it faster to seek to a position | 00:55 | |
got from 1.1s to 0.9314294s for 'abcd' x 1000000 using indexic | |||
MasterDuke | cool | 00:58 | |
samcv | going to test on non-repeat strands though | 00:59 | |
nice. from 0.9s to 0.8s :) | 01:00 | ||
10 equal length strands ( very long strands btw) | |||
3,227,579 chars per strand | |||
this is for indexic because it still uses get_grapheme_at_nocheck instead of retaining a grapheme iterator | 01:01 | ||
MasterDuke | are short strings/strands also faster? | 01:05 | |
samcv | yes | 01:07 | |
i mean regardless for every grapheme in that 10 strand file, it had to find its place again | |||
so that means moving 0-9 strands forward | |||
so it would show similarly for short ones too | |||
MasterDuke | great | 01:08 | |
samcv | trying to get rid of the loop so that it only goes once | 01:12 | |
the second while loop in MVM_string_gi_move_to | |||
01:31
committable6 joined,
quotable6 joined,
nativecallable6 joined,
coverable6 joined,
bisectable6 joined,
bloatable6 joined,
greppable6 joined,
releasable6 joined,
evalable6 joined,
unicodable6 joined
01:32
benchable6 joined,
statisfiable6 joined,
squashable6 joined
01:51
ilbot3 joined
05:21
domidumont joined,
brrt joined
05:28
domidumont joined
|
|||
brrt | good * #moarvm | 05:39 | |
06:03
KDr2__ joined
06:10
brrt joined
06:41
leont joined
06:50
domidumont joined
07:38
TimToady joined
08:01
zakharyas joined
|
|||
timotimo | good brrt | 08:21 | |
oh, i'm a bit late | |||
brrt | ohai timotimo | ||
i'm still awake :-) | |||
08:43
domidumont joined
09:13
Skarsnik joined
|
|||
samcv | good * | 09:35 | |
well i successfully got rid of the loop in MVM_string_gi_move_to that moves within a strand (or repeated strand) | 09:36 | ||
yay speedups | |||
and reduced the number of operations inside the loop that locates the strand | |||
github.com/MoarVM/MoarVM/compare/m...gi_move_to | 09:40 | ||
and removed and changed some code, added comments. since it was kind of confusing initially | 09:41 | ||
hmm i think there's a bug where if we aren't at the first position in specific strand, aka gi->pos != gi->start and it's a repetition, then it could skip too much | 10:00 | ||
strand_graphs = (gi->end - gi->pos) * (1 + gi->repetitions) | |||
this is what we have now. but i think it's wrong. strand_graphs is the number of graphs left in the strand and lets us know if we need to go to the next strand, in case we are trying to move further than there are spots in that strand | 10:01 | ||
so gi->end - gi->pos should give the space left in the current strand. but if we have repetitions, let's say gi->start is 0 and gi->end is 10. we're at 5 and there's 2 reps | 10:02 | ||
(10 - 5) * (1 + 2) = 15. but i think it should be gi->end - gi->pos + (gi->end - gi->start) * (gi->repetitions) | 10:03 | ||
10:03
zakharyas joined
|
|||
samcv | if jnthn is here let me know what you think. i think i'm right but i could be wrong | 10:03 | |
Geth | MoarVM: 5bf652edbb | (Jonathan Worthington)++ | src/spesh/log.c Don't decont native refs when spesh logging That will result in the value being boxed. This allocation could make a problem by causing a GC run to happen, but much more importantly it leads to rather misleading spesh stats that could in turn end up with selection of a bogus multi-dispatch candidate during optimization. This thus fixes the mis-specialization that has caused failures in the atomics spectests. |
10:06 | |
jnthn | samcv: Is this in move_to? | ||
samcv | yep | ||
jnthn | Wasn't the situation a while back that we only ever expected it to be called once? | ||
samcv | yeah | ||
jnthn | I suspect it's a leftover assumption from then | 10:07 | |
samcv | KMP doesn't skip more than one forward so we are currently okay | ||
okay as long as what i'm saying sounds right :) | |||
but i got ignorecase index from 0.9s to 0.8s for a 10 strand string | |||
jnthn | That is, if we weren't going from the start then we'd never be in the middle of a repetition. | ||
Nice :) | |||
samcv | 10 copes of tolstoy and repeated some times | ||
and removed the second loop | 10:08 | ||
simplified some | |||
also github.com/MoarVM/MoarVM/compare/m...50dc0efR96 | |||
can you tell me about this? | |||
in the original code, we check if gi->repetitions < remaining_reps | |||
but if the reps is less than remaining_reps, i don't get why we would set remaining_reps to gi->repetitions. that sounds like we should never get in that circumstance | 10:09 | ||
where there are'nt eneough reps left | |||
mathematically and practically unless there were corruption or something | |||
since strand_graphs should calculate there's enough space left | 10:10 | ||
or strand_graphs or whatever | |||
jnthn | remaining isn't just within this strand | 10:11 | |
So it's just saying "if the place we're going is longer than the number of repetitions, then eat all of them" | |||
(But no more) | 10:12 | ||
samcv | that should never happen though? | ||
jnthn | Thus making sure gi->repetitions doesn't go negative | ||
Thus ensuring that if (gi->repetitions) { doesn't falsely pass | |||
samcv | because we would be on the strand after it already | ||
i changed that to if (remaining) instead of if (gi->repetitions) | 10:13 | ||
since that seemed more reliable to count on | |||
jnthn | ah, you're saying we'd have skipped it above? | ||
samcv | and it should always happen at the same time. at least it never happened running roast | ||
yeah | 10:14 | ||
jnthn | Yeah, I see that now | ||
samcv | and i also reforatted it so we have no loop below :) | ||
which is pretty nice | |||
jnthn | Yeah, looks like an aubndance of caution costing us some cycles :) | ||
samcv | yeah no problem :) | ||
always better safe than sorry tbh | |||
jnthn | But yes, agree it can be simplified :) | ||
samcv | and comments!! | ||
:) | |||
nice i get full spectest pass with changes :) | 10:15 | ||
dogbert17 | trick question: is there a reason why MoarVM isn't compiled with -Wall ? | ||
10:16
robertle joined
|
|||
timotimo | all? why stop there! let's dial it up to -Wextra! | 10:16 | |
samcv | why wstop wthere? wpreface wall words with w | ||
dogbert17 | well reading the discussion above it spits out warnings like: | 10:17 | |
src/strings/iter.h:90:23: warning: ‘gi.start’ may be used uninitialized in this function [-Wmaybe-uninitialized] | |||
MVMuint32 rep_graphs = gi->end - gi->start; | |||
perhaps a false positive but there's quite a lot of stuff like that | 10:18 | ||
brrt | dogbert17: we have that voluntarily, it's called clang :-P | 10:19 | |
in seriousness | |||
i'm not against having that as a configure option | |||
dogbert17 meets a -Wall of resistance :) | 10:22 | ||
Geth | MoarVM: 6d627da28d | (Jonathan Worthington)++ | src/spesh/stats.c Handle native refs better in spesh stats Previously, we would always consider a type tuple involving one as being incomplete, since a NativeRef is a container but we don't (any more) log the type of its content (and when we did, we got it wrong by logging the boxing of the actually referenced type). This allows for generating observed type specializations of things that get passed native references. It's not enough, however, to see invocations of those be fastinvoke'd or inlined (at least, it not in the multi-dispatch case). |
10:39 | |
jnthn | Hah, yay, seems I got us inlining atomic int loads | 10:50 | |
Presumably (though will check) that means atomic int cas also inlines | 10:51 | ||
nwc10 | but first lunch? | 10:54 | |
jnthn | Well, nodelay/blocking spesh stressing | ||
But yeah, it's about lunch time | |||
Looking good, spectesting with those stress options while I go for lunch | 10:57 | ||
11:34
domidumont joined
|
|||
jnthn | yup, looking good | 12:01 | |
Geth | MoarVM: fc75bca3c9 | (Jonathan Worthington)++ | src/6model/reprs/MVMMultiCache.c Support spesh multi resolution involving nativeref This means that we can fastinvoke or inline candidates being passed a native reference. This makes, for example, the load of an atomic int possible to inline, but likely enables a lot of native inlining more generally. |
12:03 | |
12:07
committable6 joined
12:12
Skarsnik_ joined
|
|||
jnthn | brrt: The EXPR JIT works at BB level at the moment. I'm wondering if we might not get a lot more bang for buck even with that restriction if we were to make a pass of fusing basic blocks that needn't be basic blocks thanks to the elimination of invokish things and other code... | 12:12 | |
brrt | yes. | 12:13 | |
that would be pretty cool | |||
nine | So much for "no room for improvement" ;) | 12:15 | |
brrt | but then, we also need 100x more templates :-P | ||
hehe, it's more like, 'where on earth should we start' | |||
re, invokish ops | 12:16 | ||
jnthn | nine: heh, there's tons of room for improvement :) | ||
brrt | repr-specific templates, i could imagine replacing an invokish implementation with a noninvokish one | 12:17 | |
e.g. for stringification and stuff | |||
jnthn | Hurrah, cas for integers is indeed being inlined | ||
brrt | so i think, a template should carry some flag that indicates it isin't invokish | 12:18 | |
which should overrride the per-op flag | |||
we should porbably keep the per-op flag though | |||
jnthn | Aye | 12:19 | |
Many invokish ops are rewritten to non-invokish during specialization, though | |||
brrt | true | 12:20 | |
jnthn | Hm, next curiosity: why is the == in `last if cas($value, $orig, $orig + $i) == $orig` being fastinvoked but not inlined... | 12:23 | |
ilmari | huh, none of the routines in docs.perl6.org/type/atomicint appear to be indexed | 12:25 | |
jnthn wonders why | 12:27 | ||
jnthn has no idea how indexing works, though | |||
I figured it went on headings or some such | |||
ilmari | the operators are, though | ||
jnthn | Hm, even more odd | 12:28 | |
Open a docs ticket, mebbe | 12:29 | ||
Somebody probably knows :) | |||
ok, the == ends up with an assertparamcheck left behind | 12:30 | ||
brrt | and there was another thing with the multiple basic blocks… current expr trees only have a 'forward' CFG, and with multiple BBs, i need to handle backward ones as well, and that is mostly relevant for the live ranges holes finding code | 12:31 | |
jnthn | ah, yes | ||
Think I can see what's going on with the == thing | |||
Hmm | |||
Guess fixing this will make a lot of code mixing native/non-native integers happy... | |||
ilmari | ah, they're "=head1 Routines \n =head2 foo", instead of "=head1 Methods \n =head2 routine foo", which seems to be the standard | 12:33 | |
jnthn | Hurrah, the == is inlined there now too | 13:26 | |
timotimo | wowza, i might just have kicked a mid-life crisis out of the heapanalyzer's "parse snapshot" code | 13:34 | |
gc time went from 25% to 15% | 13:35 | ||
suddenly 1000 OSRs more than before | 13:36 | ||
Geth | MoarVM: ab28683b2f | (Jonathan Worthington)++ | src/spesh/optimize.c Set facts on nativeref deconts. It would, at some point, be nice to have better code-gen for the decont/box itself. But for now, we can get a notable win from setting up the facts on the target of the decont. We know the type of the result and also that it will be concrete, which in turn allows for full elimination of parameter checks in the case infix:<==>(Int, Int) ... (5 more lines) |
13:38 | |
jnthn | With that, something like | ||
loop { | |||
my int $orig = $value; | |||
last if cas($value, $orig, $orig + $i) == $orig; | |||
} | |||
Will be fully inlined | |||
(the last, cas, and ==) | 13:39 | ||
brrt | jnthn++ | ||
timotimo | i like the sound of this | ||
i shall grab latest moarvm and see if it changes things in the heapanalyzer, too | 13:40 | ||
nwc10 | in the bigger picture, what does this buy us - less CPU burned doing concurrency? | 13:41 | |
timotimo | 2x as much inlining, but also 2x as many just-interpreted frames | 13:42 | |
jnthn | nwc10: Taken together, these help pretty much any code that passes around native references | 13:43 | |
By adding various missing pieces that block inlining (or just cheapening the call by reducing duplicate checks) | 13:44 | ||
Code using the atomic ops on native ints gets a particular win though | 13:45 | ||
'cus this is exactly what it does | |||
14:04
zakharyas joined
|
|||
timotimo | now collectables are parsing faster than references (in this case there's 456_418 collectables and 1_761_280 references) | 14:07 | |
sadly, the code is full of nqp::shift_i and nqp::push_i | |||
jnthn | Hm, darn. So it seems that I can now have us re-compute the dominance tree before the second pass | 14:11 | |
timotimo | ooooh | 14:12 | |
jnthn | However, doing so causes the second pass to start descending into things it never did before. It then manages to mess things up enough to send some code into an infinite loop. | ||
timotimo | did you do the work necessary to put inlines in the "correct" spot? | ||
jnthn | No | ||
timotimo | OK, i see | ||
jnthn | That's a separate task | ||
timotimo | i still have a branch for that that i probably didn't push; it doesn't work properly, but it's a start | ||
samcv | jnthn, i confirmed it | 14:20 | |
the old code doesn't work | |||
if you move the iterator after it's already been moved | |||
in MVM_string_get_grapheme_at_nocheck i had it randomly move the iterator until it finally gets to the end destination, and i get it iterating too far | 14:22 | ||
timotimo | i find it annoying that weechat decided to make samcv, nwc10, lizmat, and brrt have the same color | ||
samcv | you can probably change it | ||
to have 256 colors | |||
timotimo | i believe i did | 14:23 | |
there's a faq entry that has the magic incantation | |||
timotimo found another big optimization opportunity | 14:29 | ||
my uint64 @result .= new(1, 2, 3, 4, 5); is 2x slower than my uint64 @resulf = 1, 2, 3, 4, 5; | |||
is 2x slower than my uint64 @result; nqp::push_i(..., 1); nqp::push_i(..., 2) ... | |||
dogbert17 | jnthn: looks as if one of your fixes has nailed RT #130855 | 14:31 | |
synopsebot6 | Link: rt.perl.org/rt3/Public/Bug/Display...?id=130855 | ||
jnthn | ooh, interesting | 14:33 | |
dogbert17 | dunno which one though | ||
14:33
brrt joined
|
|||
jnthn | Wonder how far back we could be talking :) | 14:34 | |
ilmari | bisect: sub foo () {$ = 42}; for ^2000000 { $ = foo } | 14:35 | |
bisectable6 | ilmari, On both starting points (old=2015.12 new=4b02b8a) the exit code is 0 and the output is identical as well | ||
ilmari, Output on both points: «» | |||
ilmari | it fails for me on This is Rakudo version 2017.06-198-g58900e7ba built on MoarVM version 2017.06-56-g161ec639 | 14:36 | |
timotimo | 33.92user 0.65system 0:13.36elapsed 258%CPU (0avgtext+0avgdata 848288maxresident)k | 14:37 | |
119.40user 0.83system 1:27.46elapsed 137%CPU (0avgtext+0avgdata 1188220maxresident)k | |||
Zoffix | bisect: old=58900e7ba,new=HEAD sub foo () {$ = 42}; for ^2000000 { $ = foo } | ||
bisectable6 | Zoffix, Bisecting by exit code (old=58900e7 new=4b02b8a). Old exit code: 1 | ||
Zoffix, bisect log: gist.github.com/1227bf575dca207b9f...40ad1e1cb2 | 14:38 | ||
Zoffix, (2017-08-11) github.com/rakudo/rakudo/commit/76...ed2e2fe011 | |||
ilmari | "af24ab8 Can never inline frames declaring state vars." looks like a likely candidate | 14:39 | |
timotimo | since it uses so much ram, it perhaps gets a performance penalty from swapping on my laptop | 14:41 | |
jnthn | ilmari: ah, yes, that'd do it :) | 14:42 | |
timotimo flips the order of heapsnapshots and shared data around in the heap snapshot format | 14:49 | ||
Geth | MoarVM/heapsnapshot_binary_format: 6063981424 | (Timo Paulssen)++ | src/profiler/heapsnapshot.c sprinkle comments and empty lines for readabiltiy |
14:55 | |
MoarVM/heapsnapshot_binary_format: acbbde09b8 | (Timo Paulssen)++ | src/profiler/heapsnapshot.c move snapshots to beginning of file this way we can stream out the snapshots every time we GC and then free their memory, then later just finish it up with the string heap, types, and static frames. |
|||
timotimo | that was easy, wow. | 14:56 | |
15:04
travis-ci joined
|
|||
travis-ci | MoarVM build errored. Jonathan Worthington 'Support spesh multi resolution involving nativeref | 15:04 | |
travis-ci.org/MoarVM/MoarVM/builds/272027364 github.com/MoarVM/MoarVM/compare/6...75bca3c9a2 | |||
15:04
travis-ci left
|
|||
timotimo | timed out on one of the mac builds ^ | 15:05 | |
timotimo notices that all of this was just because he wanted to see if the describe functions for the new spesh structures are correct | 15:09 | ||
isn't yak shaving fun | |||
Zoffix chuckles that google's result for "wiki Yak Shaving" leads to a page that says "Wikipedia does not have an article with this exact name. Start the Yak shaving article" | 15:14 | ||
jnthn | :P | ||
15:14
zakharyas joined
|
|||
japhb | Zoffix: :-D | 15:23 | |
jnthn | Of course, the place where the set elimination blows up would be in a frame with 14 inlines :P | 15:28 | |
timotimo | my advice: just throw it out completely | 15:36 | |
have you tried the spesh step-by-step tracing gdb script i built, jnthn? | |||
jnthn | Not yet | 15:40 | |
15:47
lizmat joined
|
|||
jnthn | So the interesting thing about this case is that at the end of an inline we have | 15:57 | |
sp_fastinvoke_o r204(3), r205(2), liti16(2) | 15:58 | ||
set r206(2), r204(3) | |||
[Annotation: FH End (12)] | |||
set r123(10), r206(2) | |||
[Annotation: Inline End (5)] | |||
goto BB(1121) | |||
Being rewritten into | |||
sp_fastinvoke_o r123(10), r205(2), liti16(2) | |||
[Annotation: Inline End (5)] | |||
goto BB(1121) | |||
(yes, the FH end moves up above the fastinvoke, it's fine) | |||
Where r123(10) is outside of the inline's register set | |||
(It's the caller register) | 15:59 | ||
timotimo | right. at first glance that looks like a decent optimization (unless of course it doesn't update the facts properly?) | ||
jnthn | Don't think facts come in to it. This is the only one of the 3 set elimination transforms that actually *doesn't* screw the facts | ||
The other two do | 16:00 | ||
timotimo | OK | ||
jnthn | Which will have to be fixed later | ||
timotimo | how does it make things explode? | ||
jnthn | I don't know. My first guess is deopt | ||
timotimo | one of the registers that are being skipped were used post-deopt at that point? | ||
jnthn | The end result is an infinite loop of some sort | ||
timotimo | like, we deopted into the one that would return a value from a register that was no longer being set | 16:01 | |
jnthn | ah, yeah | 16:05 | |
For the topmost uninlinee we do this: | |||
MVMuint16 orig_reg = (MVMuint16)(f->return_value - f->work); | |||
MVMuint16 ret_reg = orig_reg - cand->inlines[i].locals_start; | |||
uf->return_value = uf->work + ret_reg; | |||
timotimo | we grab the contents of the register that used to get used for the return in question? | 16:06 | |
jnthn | Note the substraction | ||
orig_reg - cand->inlines[i].locals_start | 16:07 | ||
That'll become *negative* | |||
And point to...heck knows where | |||
timotimo | oooh, ouch | ||
because we changed the register in the op | |||
d'ouch | |||
jnthn | Right | ||
jnthn ponders the best way to deal with this | 16:08 | ||
timotimo | i can't believe we didn't have a string constant for "filename" or "path" or something yet | 16:12 | |
jnthn | I guess any invokish is implicated | 16:16 | |
Oh, maybe not | |||
Hmm | |||
Yeah, actually it perhaps is. | |||
I guess the boring answer is to just not allow a rewrite to the outside of the current inline | 16:17 | ||
timotimo | hm, right | 16:18 | |
jnthn | Though we don't immediately have the info available to do that | 16:19 | |
(We don't store *which* inline a basic block maps to in a convenient way) | |||
Ah well, think I might leave it for tomorrow, getting a tad tired now | 16:20 | ||
timotimo | sure thing | ||
good work so far! | |||
we can just toss the rewriting opt for now ;) | |||
jnthn | Well, we need 'em in the end, so... | 16:21 | |
The reason I set off trying this was that I wanted to move the exception handler -> goto opt to be post-inline | |||
Spotting that it would apply to the cas loop but also to other things | |||
But since we didn't have the inlines in the dominator tree, well, we'd not get it | 16:23 | ||
Geth | MoarVM: 9a5987f030 | (Jonathan Worthington)++ | src/spesh/optimize.c A little refactoring of second optimization pass No functional changes, just preparing it for doing a bit more, and breaking the code into easier to reason about pieces. |
16:24 | |
MoarVM: 73bfaf414f | (Jonathan Worthington)++ | src/spesh/optimize.c Remove fact copy we most likely don't now need Since propagating information during the optimize phase is so critical to getting more inlines and so forth, we'll already have done this by the second phase. |
|||
MoarVM: 3271cd8d51 | (Jonathan Worthington)++ | src/spesh/optimize.c Re-flow and better explain set elimination code Done as part of understanding its shortcomings. Also comment on the places where it needs to be improved to not result in a busted spesh graph. |
|||
jnthn | Those are some updates I've done along the way :) | ||
Will hold the other bits in reserve until tomorrow, when I figure out how to solve the inline issue. | 16:25 | ||
lizmat | jnthn: does MoarVM / libuv have some knowledge about currently open files? | ||
jnthn | No | 16:26 | |
lizmat | ok | ||
jnthn | I mean, technically MoarVM can walk the entire heap | ||
And find every file handle | |||
Which is how the heap profiler and GC do things | |||
lizmat | well, I was more thinking along the line of an array indexed on native-descriptor | 16:27 | |
jnthn | But there's no other way of tracking things | ||
No, there isn't that | |||
lizmat | a .close would clear the entry, on exit it would close all the entries in the list of open filees | ||
it still wouldn't close on scope exit like in Perl 5, but at least it would ensure buffered output would not be lost | 16:28 | ||
timotimo | the next step for the heap snapshot format would be to have pieces of strins/types/staticframes before or after every single snapshot, but only containing the difference. that way, even a ctrl-c'd program would give usable & useful data | ||
jnthn | It's just hiding a problem though, which can eventually manifest itself in too many open files. | ||
lizmat | jnthn: which could be important in chasing bugs causing programs to exit | ||
jnthn: true | 16:29 | ||
jnthn | If you're writing such a log, just open it with :!buffer | ||
geekosaur | yeh, there's a reason C makes sure stdio filehandles are recorded for atexit() to flush. I'm thinking something similar, at least for buffered handles, would be smart | ||
jnthn | We already are doing it for stdout/stderr | ||
geekosaur | if you are writing a log from a program that crashes, you lose diagnostic information | ||
and disabling buffering makes that logging expensive | 16:30 | ||
jnthn | True | ||
lizmat | jnthn: how is it done for stdout/stderr ? | ||
jnthn | Because those handles are created by the VM and hang off ->instance | ||
We can of course keep an array of all open files, and remove stuff fromit | 16:31 | ||
So long as you accept that the GC will then *never* close a file for you if you forget, so every opened file hangs around until program exit | |||
geekosaur | if it isreally hard /expensive to do for all handles then maybe a new parameter to add to an autoflush list would be an idea | ||
jnthn | I could live with that trade-off I guess | ||
In that it will also make resource-leaky programs fall in a heap faster | 16:32 | ||
geekosaur | but really I think the 'hangs around' issue will likely force action on this at some point | ||
jnthn | geekosaur: 'hangs around' issue? | ||
16:32
travis-ci joined
|
|||
travis-ci | MoarVM build passed. Jonathan Worthington 'Set facts on nativeref deconts. | 16:32 | |
travis-ci.org/MoarVM/MoarVM/builds/272058760 github.com/MoarVM/MoarVM/compare/f...28683b2f2f | |||
16:32
travis-ci left
|
|||
geekosaur | doesn't get closed automatically when unreachable | 16:33 | |
jnthn | geekosaur: Ah, you're saying that if we never do that, then people will be forced to fix their stuff? | ||
geekosaur: Or that this would count as a bad behavior? | |||
geekosaur | because if you can't currently do it except by explicit close then things kinda get nasty. I don't think we quite have Haskell's issue there (basically Haskell *must* close stuff when it becomes unreachable, because laziness means you otherwise have no rel control over how many open handles you have) | ||
but if making an expression lazy can cause this then yes, I foresee problems in the future | 16:34 | ||
jnthn | I don't really see, short of the huge yak shave involving implementing weak references, how we can both have close on unreachable *and* keep a list of open things to close at exit. | ||
geekosaur | of course we don't have Haskell's style of lazy I/O either (unsafeInterleaveIO is convenient but has a ton of traps associated with it) | 16:35 | |
lizmat | my view: | ||
1. you should have something like a my $handle will leave { .close } | 16:36 | ||
geekosaur | basically I am envisioning a situation where some object has a handle embedded deep within it and eventually becomes unreachable. of course that's begging for EMFILE issues anyway, but sometimes it's the best option for doing some things | 16:37 | |
lizmat | 2. if you don't, you will eat up resources wrt to number of open handles | ||
3. if you exit, MoarVM should properly close all the open file handles before exitin | |||
jnthn | Well, there's the two of you wanting the opposite solution :P | 16:38 | |
geekosaur | jnthn, actually I am thinking handles could go in a separate gc "bucket" so that at exit only that bucket could be gcd | ||
Zoffix 's most common IO doesn't touch handles at all... | |||
geekosaur | and only gcd for the handles not for general objects, but that may be special casing things a bit too much | 16:39 | |
jnthn | It is. | ||
The GC only has two spaces, nursery and gen2 | 16:40 | ||
lizmat | in my opinion, the damage from losing buffered data is much larger than the damage of too many open handles | ||
Zoffix | Make that bucket more generic, so you could have Perl's equivalent of DESTROY that's guaranteed to run on exit | ||
class Foo is Collectable {... } | |||
Zoffix is talking from complete ignorance of GC stuff, mind you :) | |||
jnthn | lizmat: Note that you can implement what you're suggesting in Rakudo | 16:41 | |
lizmat | jnthn: yes, and I'm tempted :-) | ||
which would solve the issue also for other backends, I guess :-) | 16:42 | ||
jnthn | Indeed. | ||
And means it's more clear for others to see what happens | |||
Also | |||
If at some point we do get weak references, then we can have our cake and eat it | |||
(By making the open handles list be a list of weakrefs, so they won't keep handles alive if nothing else would) | |||
If we do it at Perl 6 level we also get the chance to make the feature opt-in at Perl 6 level should we need to in the future. | 16:43 | ||
As well as making it easier to build features like "store a stack trace with each open if an env var is set, and at exit show me all the handles I leaked" | 16:44 | ||
lizmat | ok I'll put that on my list | 16:46 | |
jnthn | Alright | ||
lizmat | meanwhile, I need to decommute& | ||
geekosaur | mrrr. and now I am thinking 'make that resources instead of just handles' :p | ||
jnthn | fwiw, weak refs are possible, just...need notable effort | ||
geekosaur | (ResourceT in Haskell) | ||
jnthn | Step 1 being going to re-read the appropriate chapter of the GC handbook so I do it in a sensible way rather than re-inventing a poor one :) | 16:47 | |
When I may find it's not quite as bad as I fear... | |||
But even then we need to figure out how we'll expose it at Perl 6 level, and if we need things like a callback when the thing gets collected | 16:48 | ||
(Otherwise, you never reclaim your array slot) | |||
And all that fun | |||
Probably in the medium term, it's what we want though | 16:49 | ||
[Coke] | jnthn: Looks like 1.0.2 is the latest version in macports of openssl, btw. | 16:56 | |
jnthn | [Coke]: Yeah, the ALPN love is going to be slowly shared, it seems... | 16:57 | |
Skarsnik | hm I replaced github.com/MoarVM/MoarVM/blob/mast...all.c#L810 with cptr = (void*)((uintptr_t)storage + (uintptr_t)repr_data->struct_offsets[i]); | 17:02 | |
and it make efence happier | |||
[Coke] | oops. wrong window, sorry | 17:05 | |
jnthn wanders home | 17:08 | ||
17:26
zakharyas joined
|
|||
timotimo | fun, when a heap snapshot file has been truncated by a ctrl-c'd process, it won't have the index at the end | 18:22 | |
unless i write out a full index after every bit, that would work, because there'd always be an index at the end | 18:27 | ||
geekosaur | and here you see why I mentioned ResourceT earlier :p | 18:29 | |
18:48
dogbert2 joined
19:31
zakharyas joined
19:46
lizmat joined
19:58
ggoebel joined
20:01
greppable6 joined
|
|||
timotimo | this is C-level :) | 20:08 | |
Skarsnik | it's me or github.com/MoarVM/MoarVM/blob/mast...all.c#L746 and github.com/MoarVM/MoarVM/blob/mast...all.c#L795 could probably be merged? | 20:55 | |
geekosaur | are the masks etc. all guaranteed to be the same? | 20:56 | |
Skarsnik | I will look at that tommorow x) | 20:57 | |
Geth | MoarVM/heapsnapshot_binary_format: a8faddedd7 | (Timo Paulssen)++ | 5 files WIP on streaming snapshots to file on every gc |
21:28 | |
timotimo | (just for transferring to my desktop) | 21:29 | |
MasterDuke | timotimo: i tested it a tiny bit yesterday. took 25s to `summary` a snapshot of `perl6 -e ''` | 21:31 | |
yoleaux | 06:19Z <AlexDaniel> MasterDuke: maybe you have some idea. committable.t and benchable.t hang on the first “Did you mean …” test | ||
06:23Z <AlexDaniel> MasterDuke: sorry, bisectable.t and benchable.t. So github.com/perl6/whateverable/blob...ble.t#L165 and github.com/perl6/whateverable/blob...ble.t#L142 | |||
06:23Z <AlexDaniel> MasterDuke: it may be associated with Sift4 module, which is problematic by itself… for example, it's still leaking memory. | |||
MasterDuke | which i thought was a long time, so i timed the old heap analyzer, 100s for the same thing | ||
i just usually go do something else and didn't realize exactly how long it did take before! | 21:32 | ||
timotimo | MasterDuke: how big is the file, how good is the multi-core utilization? | 21:33 | |
also, which commit did you test with? i made things even faster still | 21:34 | ||
MasterDuke | 22mb, don't remember exactly, but think i was seeing ~315% cpu in top | ||
timotimo | that's not so bad | ||
MasterDuke | 16d9c2220187018410ae6a482398920b170e0a07 i think | 21:35 | |
timotimo | you should be able to check out the last commit before head and try again | ||
MasterDuke | cool, i'll do that sometime this evening | 21:36 | |
timotimo | nice | 21:38 | |
21:39
AlexDaniel joined
|
|||
timotimo | the next changes i'm making aren't about making things faster, i'm afraid | 21:39 | |
they will make running a program with the heap snapshot profiler active a bit cheaper on memory, though | |||
MasterDuke | that's good too | 21:40 |