00:01 unicodable6 joined
samcv jnthn, i think I have made all the changes you requested with my PR now 00:25
moved the renormalized_section_graphs = MVM_string_graphs later and added documentation for the return values of should_break
00:29 lizmat joined 01:00 greppable6 joined, quotable6 joined, unicodable6 joined, bisectable6 joined, benchable6 joined, coverable6 joined, committable6 joined, evalable6 joined, bloatable6 joined, statisfiable6 joined 01:06 stmuk_ joined 01:49 ilbot3 joined 02:00 lizmat joined
Geth_ MoarVM: c814c79e1f | (Samantha McVey)++ | src/strings/ops.c
Deduplicate code from MVM_string_indexing_optimized

The code is repeated in collapse_strands so call that function instead of repeating the code.		 02:03
MoarVM: samcv++ created pull request #616:
Start off collapsing strands as 8bit instead of 32bit		 04:48
05:40 brrt joined
Geth_ MoarVM/even-moar-jit: df71570c04 | (Bart Wiegmans)++ | 7 files
Make even-moar-jit buildable on MSVC and nmake

nmake is why we can't have nice things		 06:02
brrt so, that makes it at least buildable
oh, hang on 06:19
wrong define syntax 06:20
06:27 TimToady joined
Geth_ MoarVM/even-moar-jit: af6ba43e9c | (Bart Wiegmans)++ | Configure.pl
Fix define typo in dynasm flags

Unlike c compilers, dynasm understands -D *space* name=value, and doesn't understand them otherwise		 06:28
06:46 brrt joined 06:52 domidumont joined 07:00 domidumont joined
samcv hey brrt 07:09
brrt ohai samcv 07:11
shall we join in bashing nmake? 07:12
samcv sure
that's the windows make utility right
brrt yeah, well, it's the one on the MS toolchain 07:14
it is
a): non-parallel
samcv yeah i remember it having "issues"
brrt b): doesn't support inference in a 'standard' way
c): doesn't have the $< defined if it *isn't* working on an inference rule 07:15
anyway
how are you?
nwc10 good *, * 07:18
Geth_ MoarVM/even-moar-jit: 838aa30fd7 | (Bart Wiegmans)++ | 3 files
Add 'include:' keyword to expr template compiler

This is mostly so we don't have to redefine macros when we start making specialized templates for REPR types.		 07:20
brrt good hi, nwc10
07:33 zakharyas joined
nine Couldn't we just ship a working make for Windows? 07:39
brrt you know.. that's not actually such a bad idea 07:45
nine I think the one good thing about Windows is that you can ship binaries with a reasonable chance of them actually running on the target system. 07:47
And now that I think of it, this would probably be a good idea for Distribution::Builder::MakeFromJSON, too.
brrt otoh, a pure-perl 'make' would probably not be the hardest program in the world 07:48
samcv brrt, oh i'm good :) 08:17
08:38 robertle joined, domidumont1 joined
brrt something is horribly broken about that patch though 08:40
with the macro stuff
jnthn morning o/
nwc10 jnthn: origin/spesh-worker^ got all the way through to running the rakudo spectest 08:41
jnthn Yup, because HEAD more fully re-enabled it and now we can see the problems in all their glory :) 08:42
samcv hey jnthn
08:48 domidumont joined
jnthn o/ samcv 08:50
Your PR looks good, just noted one tiny thing :)
jnthn goes to take a look at the guard tree issue 09:03
wat, it gives two differnet errors and works under valgrind o.O 09:06
Geth_ MoarVM/spesh-worker: 2a2f0b2bb0 | (Jonathan Worthington)++ | src/spesh/arg_guard.c
Fix inverted if/else branches.

Should use the safepoint mechanism to release guards when asked.		 09:12
jnthn Now it reliably explodes in the way I understand. :) 09:15
brrt hehe 09:17
reliable explosions++
jnthn ponders the best way to fix this
nine Also brauch ma ka Angst haben, dass ihm nix mehr einfaellt ;) 09:27
oops, ww 09:28
jnthn Nothing to have angst about :P
...is "Also" really German? :)
Also, phew, think I found a fix for my problem that is both (a) the easier one to implement without screwing lots with the graph, and (b) potentially more useful too :) 09:29
samcv jnthn, what did you mean by "I wonder to what degree we can take the inputs being 8-bit as an indicator that this is likely to work out? " 09:30
jnthn samcv: Can we use the strings the strands reference all being themselves 8-bit ones as a heuristic up-front? 09:31
Or do we have so many 32-bit ones floating around that need not be 32-bit that there's no point?
Or more generally, can we find a good/cheap heuristic to tell us whether to try the 8-bit thing at risk of having to copy later 09:32
samcv well
nine jnthn: yes, it means "thus"
or just "so" 09:33
jnthn Ah, so not a modern English loan after all :)
samcv well 8 bit is done easily, and 32bit it only has to trace through as much as the 8 bit takes to trace through
so i think it should be faster in most cases. i gave the worst case in my timings with the unicode at the very end of the really long string
but as far as heuristics hm 09:34
jnthn, how does perl6/nqp create its own strings?
since i don't know. i know how to do ops to existing strings but like. when it reads a source file how does it create those strings 09:35
with what ops
i mean we'd have to set a flag when these strands are made or whatever 09:36
and check every codepoint
jnthn It will just substr the source to get the string text. String literals mostly live in QAST::SVal nodes. These are turned into MAST::SVal nodes.
These are in turn stored in the string heap inside of the assembled bytecode
Even when we compile "in memory" rather than writing it out, we go through this process 09:37
Strings are deduplicated here: github.com/MoarVM/MoarVM/blob/mast...ler.c#L291 09:38
And then formed into the bytecode string heap here: github.com/MoarVM/MoarVM/blob/mast...er.c#L1319
Either as UTF-8 or latin-1: github.com/MoarVM/MoarVM/blob/mast...er.c#L1343 09:39
The flag saying which they are is used to pick the MVMString representation (8 or 32) when we load the bytecode
Though we also lazily deserialize the string heap 09:40
Since many strings in CORE.setting are exception messages and so never used, for example :)
samcv ah 09:49
but the strings themself don't have any flag i guess and we should add one? 09:56
10:01 TimToady joined
jnthn samcv: Well, we know what kind of string they are 10:02
samcv: Question is if we're good enough at sticking at producing 8-bit strings when we get them as input
If so, then we could use the kind thing as the flag
Geth_ MoarVM/spesh-worker: 9823340640 | (Jonathan Worthington)++ | src/spesh/arg_guard.c
Correct a typo.
 10:14
MoarVM/spesh-worker: c34ce8315b | (Jonathan Worthington)++ | 2 files
Handle certain specializations in the guard tree.
MoarVM/spesh-worker: cf35c88bbe | (Jonathan Worthington)++ | src/spesh/log.c
Fix duplicate free and mutex destroy.
jnthn Hm, somewhere it seems we return a real NULL 10:18
With a local patch for that, things are back on their feet again 10:21
Geth_ MoarVM/even-moar-jit: 63 commits pushed by (Daniel Green)++, (Jonathan Worthington)++, (Samantha McVey)++, (Stefan Seifert)++, MasterDuke17++, (Steve Mynott)++, lizmat++, (Lucas Buchala)++, (Timo Paulssen)++, (Bart Wiegmans)++
review: github.com/MoarVM/MoarVM/compare/8...3fa4f9d974		 10:39
brrt also sprach nine 10:40
Geth_ MoarVM/spesh-worker: ac895f243e | (Jonathan Worthington)++ | src/spesh/arg_guard.c
Cope with result after callsite test.
 10:45
jnthn With that, NQP builds/tests with MVM_SPESH_BLOCKING=1 (which means we never get lucky and avoid a bug by doing less spesh work due to exit) 10:47
Geth_ MoarVM/spesh-worker: 9e072734f8 | (Jonathan Worthington)++ | src/spesh/log.c
Guard against NULL result.
 10:57
MoarVM/even-moar-jit: 0a91534921 | (Bart Wiegmans)++ | tools/expr-template-compiler.pl
Undo a thinko in expr-template-compiler

The compiled template is flattened into the shared buffer, so we need the offset before pushing.		 11:09
MoarVM/spesh-worker: 740ed95f85 | (Jonathan Worthington)++ | src/spesh/candidate.c
Remove now-unused function.
 11:11
MoarVM/spesh-worker: 4811d45bce | (Jonathan Worthington)++ | 7 files
Remove hard limit on spesh candidate count.
 11:20
jnthn lunch & 11:22
11:36 geekosaur joined 11:59 zakharyas1 joined
nwc10 jnthn: Stage parse : 721.273 12:01
was about 1000 on the commit(-before-last) overnight
and about 500 for a regular ASAN build
although I've set your new env variable, so this may not be a fair comparison 12:02
into spectest... 12:03
jnthn Yeah, we're still not using a bunch of the logged data yet 12:07
Now that things seem to be basically working again, I'll integrate the new logging with the graph 12:08
Geth_ MoarVM/spesh-worker: bc5992ea6b | (Jonathan Worthington)++ | src/spesh/arg_guard.h
Fix typo; MasterDuke17++.
 12:09
MoarVM: 99cba547f8 | (Samantha McVey)++ | 3 files
When concat needs normalization only normalize one section

Previously if the last grapheme of string a and the first grapheme of string b determined that renormalization was needed to combine these codepoints, we were forced to renormalize all of string a and all of string b, together into one string.
... (22 more lines)		 12:10
MoarVM: 7044f99eaf | (Jonathan Worthington)++ (committed using GitHub Web editor) | 3 files
Merge pull request #615 from samcv/concat-magic

Don't re_nfg string a + string b when concatenating in many cases, instead add only the section needing renormalization in as another strand
12:33 zakharyas joined
Geth_ MoarVM/spesh-worker: 89e597a201 | (Jonathan Worthington)++ | 3 files
Pass planned candidate info into facts.

This will be used to find logged values.		 12:34
MoarVM/spesh-worker: 60864219a4 | (Jonathan Worthington)++ | 4 files
Add logged op annotation.

For ops that log something, but that aren't then used as deopt points. Will be used to save a bytecode index annotation in the spesh graph so that we can resolve logged values for static lexicals.
MoarVM/spesh-worker: d2d7652f9a | (Jonathan Worthington)++ | 3 files
Insert logged annotations into spesh graph.

Carrying bytecode position of an instruction marked :logged.		 12:48
MoarVM/spesh-worker: 5fdf96ddbf | (Jonathan Worthington)++ | src/spesh/graph.c
Fix comment grammar.
13:03 domidumont joined
jnthn language class & 13:21
nwc10 jnthn: it survived to the end of the spectest 14:08
14:09 brrt joined 14:11 zakharyas1 joined 14:21 brrt joined
jnthn nwc10: And failed a handful, perhaps? 15:10
15:11 domidumont joined
jnthn bah, my pre-lesson bug hunting was actually bogus, I think 15:23
15:24 brrt joined 15:30 lizmat joined
brrt wow, jit-bisect works on windows 15:30
all but the comparify 15:31
jnthn Nice :)
brrt i do have strawberry-perl installed, so that probably explains the pressence of objdump
Geth_ MoarVM/spesh-worker: afedbe4092 | (Jonathan Worthington)++ | src/spesh/graph.c
Logged annotations should have pre-increment PC.
 15:34
MoarVM/spesh-worker: fef46d93be | (Jonathan Worthington)++ | src/spesh/optimize.c
Resolved logged static values.
 15:36
15:49 colomon joined
timotimo jnthn: that's a nice infinite loop you have there 15:56
while (ann) and not changing ann inside the loop 15:57
jnthn heh, nice catch :) 15:59
15:59 lizmat joined
jnthn Apparently nowhere do we have multiple annotations on such a node :) 15:59
Geth_ MoarVM/spesh-worker: 8f0afa0e09 | (Jonathan Worthington)++ | 6 files
Ensure spesh log has correct thread.

Even when it's created by a thread other than the one its for. Prior to this fix, we filled all the spesh logs, sent them off to the spesh worker thread, which then after specializing created a log with the wrong thread object in it. The upshot was that we never created any more logs after that one for the logging/sending thread.		 16:03
MoarVM/spesh-worker: 6e3b3cb9ef | (Jonathan Worthington)++ | src/spesh/optimize.c
Fix potential infinite loop; timotimo++.
 16:04
16:13 zakharyas joined 16:16 colomon joined
jnthn Seems that at the moment we're managing to consume huge amounts of time making specializations, perhaps due to a lack of limit on how many, perhaps due to some other kind of bug. Will need to dig a bit more. 16:22
I should re-instate the hotness thresholding too 16:23
I had thought that maybe since we're on a separate thread then we can just produce specializations without worrying about code size 16:24
But when things have 500 basic blocks, well, no :P
Not unless it's getting really hot
16:39 robertle joined
timotimo ah yeah, there's really big frames 16:47
i think find_best_dispatchee for example is gigantic
Geth_ MoarVM/spesh-worker: 9bcbc02cb4 | (Jonathan Worthington)++ | 2 files
Log time that specialization work consumes.
 17:04
jnthn Dinner time; bbl
17:39 domidumont joined 17:48 brrt joined 17:54 domidumont joined 18:10 vendethiel joined
nwc10 jnthn: IIRC yes failed a few, but I'm not totally sure 18:27
(which)
19:11 domidumont joined 19:34 dalek joined 19:44 SourceBaby joined 19:48 synopsebot6 joined 20:06 tinyurl_comSLASH joined 20:08 tinyurl_comSLASH left
timotimo hm. so we have a deopt point in front of a p6oget_o, right? and that keeps the wval for the type object we're accessing an attribute for alive 20:41
but can the code actually deopt in a way that puts us in front of the p6oget_o in the original code so we actually reqiure the wval still? 20:42
21:04 AlexDaniel joined
timotimo i don't know enough about deopt to figure out how we could robustly realize a variable isn't used any more after a given deopt point 21:15
jnthn: will we have something in place that helps us inline more stuff when the containing frame has already been speshed before the potential inlinees have been speshed sufficiently? 21:27
we don't really have the "spesh on the way up the callstack" any more, or does the lagging approach still somehow do that? perhaps if we do the log in reverse?
jnthn timotimo: We sort the plan in order of maximum call depth 21:37
timotimo ah, that makes sense
i was wondering why that was there
21:37 committable6 joined, bisectable6 joined, benchable6 joined, coverable6 joined, bloatable6 joined, unicodable6 joined, greppable6 joined, evalable6 joined, quotable6 joined
jnthn On deopt points - they're after a fetch operation, and guard the result of the fetch 21:38
So anything involved in the fetch that's not otherwise needed, like a wval, can go away
They guard is there because something that uses the fetched value expects it to be of a certain type 21:39
timotimo but in this case there isn't a guard? 21:42
the last steps are wval, getattr_o, return_o
jnthn Hm, do you have a snippet of spesh log? 21:43
timotimo one sec
jnthn There is some amount of usage bumping we do in order to make sure stuff is available to deopt'd code though 21:44
timotimo gist.github.com/timo/3a3bdd6a6b7c9...5076eec34f
yes
that's what i'm blaming here
thing is, that the post-deopt code doesn't use the value any more
jnthn Oh, I see what you meant 21:46
*mean
timotimo yeah i still mean it :P
jnthn I think the mechanism is probably conservative rather than precise, but this does seem like an obvious case 21:47
It could be off-by-one-ish
Oh, also
timotimo i first thought that, too. that we compare >= the idx of where the deopt is rather than >
jnthn I wonder if we attach usages to deopt points that bumped them
So we can decrement them if the guard goes unused 21:48
timotimo i don't think we store that at all
jnthn I suspect that needs a good bit of care to get right
But I guess the dominance order gives us some good properties there
timotimo it should, though i don't think we properly restore dominance order when inlining has happened
so need extra care there likely
i wonder if it'd be worth it at all to turn wval into spesh slot lookup? 21:50
jnthn I don't think so 21:52
After JIT they are equivalent
timotimo ah, that's good 21:56
it's probably not the juiciest target for non-jitted frames, i guess 21:57
jnthn True, though every little helps :) 22:00
timotimo i think we should be able to use "perf script" to get per-opcode stats rather than just "20% time spent in MVM_interp_run" 22:01
jnthn urgh, so warm weather this evening :/ 22:10
Doubt I'll sleep but don't have the energy to do anything either.
timotimo i know that feeling very well 22:13
i just installed a fly screen in one of my windows and it's reasonably cool outside
jnthn Hopefully it'll cool a bit more here... 22:22
Well, guess I'll go and lie down anyways... Hopefully tomorrow will bring a storm to cool things off 22:23
o/
timotimo gnite jnthn! 22:27
i wish you the best of luck 22:28
samcv timotimo, what would you think about a low memory mode where we don't collapse any of the strands unless we have to 22:35
like optimized indexing operation wouldn't flatten the strands, to avoid spikes of memory usage. Zoffix was saying that he had memory usage concerns when trying to use p6 for the web 22:36
Zoffix s/concerns/problems/; 22:38
As in, the app nomed too much RAM is why I switched away from Perl 6 and not that it was too slow
samcv and also for example when i had to collapse strands, it creates an 8bit string allocates it. and then if it needs to go to 32 bit it allocates the 32bit string on top of the 8 bit. 22:39
there is a way for me to deallocate the 8 bit string. but i did not
because it has a performance impact that can be somewhat big
maybe not *that* big. but if you cared more about memory usage then it's a tradeoff the user could make
but i did that to avoid performance issues with if it finds a unicode character at the very end of a long string. but i'd be open to a low memory mode that would deallocate it to avoid having to have them both allocated at once, and not collapse strands at all unless we reach the max strand limit 22:40
timotimo i was wondering at some point that we could probably start creating an 8bit string and when we hit the first big codepoint we see how far we've come and maybe build two strands instead. the first one 8bits, the second one 32bits.
samcv yep 22:41
same here. though at the moment uh. where would we do that
timotimo it'd be cool if we could be smart about "where in this string are we currently 'working'"
so we could have an expanded part "in the middle" and strands in front and in the back
samcv what function creates strings again? is there a general purpose one
timotimo but mutating a string in-place is very bad
all the decode functions from our encoders 22:42
samcv ah
ok
timotimo, what other places to we need improvements in memory usage? 22:51
timotimo depends on the workload you're interested in analyzing 22:52
like, do you want to make the memory usage of perl6 -e '' smaller?
or are you trying to parse a 10 gigabyte log text file with a grammar? 22:53
are you doing arithmetic with numbers with 1m digits?
samcv i don't know. i just meant it as a general thing if you had anything in mind 22:54
timotimo oh
i know of a few things that are mostly really tiny and inconsequential
samcv :P 22:55
i mean where does our memory go in general
timotimo fire up heaptrack to figure that out i guess
a lot of startup memory usage is from deserializing objects and code
samcv i mostly meant during runtime 22:56
i think that's what Zoffix was referring to?
timotimo can't tell without a particular workload i'm afraid
Geth_ MoarVM: dwarring++ created pull request #617:
attempt at a better error message for RT#126341		 22:57
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=126341
timotimo i have a little example script here that does some fibonacci and it allocates the majority of stuff in optimize_bb 22:59
but the backtraces are *weird* 23:00
maybe the "<unresolved function>" entries are somehow to blame?
but the optimize_bb stuff is rather spiky so that's good. 23:02
hmpf. by running heaptrack against moar as binary instead of bash (because of the launcher script) i get vastly different results 23:04
oh, i forgot to set the env vars i had in the other one
it does look like something upsets heaptrack, perhaps it doesn't work terribly well with exec? 23:19
23:28 MasterDuke joined
timotimo will go to bed a little bit earlier today 23:42
maybe tomorrow i'll add a "spent %x bytes" output to the spesh log, because that interests me
MasterDuke timotimo: hopefully quick question. could spesh info be added to pre-comp bytecode? 23:43
yoleaux 23:22Z <Zoffix> MasterDuke: This commit isn't working because $error is BOOTException, not a Perl 6 exception and it don't got .message. We need some sort of a different solution. github.com/perl6/nqp/commit/859b44...2d45a0bfd2
timotimo what do you mean? 23:44
MasterDuke so it could start out running already speshed 23:45
timotimo oh 23:46
no, not really
MasterDuke thought that might be the case. oh well, later then... 23:47
timotimo the "certain" optimizations will get us close, though 23:48
much of spesh relies on having pointers to type objects and such 23:49
not sure if we'd be able to properly write stuff like that out, for example by relying on objects that are in a serialization context and haven't been modified in any way? i guess? 23:50
anyway, seeya!

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!