github.com/moarvm/moarvm | IRC logs at colabti.org/irclogger/irclogger_logs/moarvm
Set by AlexDaniel on 12 June 2018.
Geth MoarVM/mvmeventsubscribe: c0523a1077 | (Timo Paulssen)++ | 11 files
Introducing VM Event Subscription

nqp::vmeventsubscribe lets you set a ConcurrentBlockingQueue that will receive events from the VM internals.
Currently, you can only configure it with nqp::hash("gcevent", type) where type is a int64 array, or null to unsubscribe from this kind ... (9 more lines)		 00:22
timotimo oops my hand slipped on the keyboard
00:40 evalable6 left 00:43 lucasb left 00:44 evalable6 joined
Geth MoarVM/mvmeventsubscribe: 4 commits pushed by (Timo Paulssen)++ 00:47
timotimo gist.github.com/timo/b1986b4b91cff...e279a14c6d - japhb, check it out!
that'll want to be implemented in Telemetry as well, of course 00:58
so that it'll just show "number of GCs since last snapshot" and such
japhb timotimo: Excellent! 01:21
jnthn: Full GC runs would make sense to me. I like the idea of getting Log::Timeline support for that. (Which would probably get me using Log::Timeline more. :-) ) 01:22
japhb can't wait for PEA and nativeref lowering improvements 01:23
timotimo very looking forward to if you can figure stuff out using this little feature 01:26
also, i wonder if the "start of gc" should have "start of VM" subtracted from it
japhb timotimo: That makes sense. Helps to compare runs that way. 01:33
Geth MoarVM/mvmeventsubscribe: 465bb2dccb | (Timo Paulssen)++ | src/moar.c
don't forget to unlock before returning
 01:44
01:50 zakharyas joined
Geth MoarVM/mvmeventsubscribe: 795d523739 | (Timo Paulssen)++ | 3 files
vmevent: subtract vm startup time from gc start time
 01:59
timotimo may also want to divide the numbers by 1000 before sending them off
because other places, like the profiler, divide the values from uv_hrtime by 1000
02:21 Kaiepi joined 02:44 robertle_ joined 02:47 robertle left 04:54 robertle joined 04:57 robertle_ left
nine timotimo: notification about GC happening could make DESTROY related tests more reliable. When we detect that a GC happened but DESTROY was not called, there's obviously an issue while even with nqp::force_gc we cannot be sure that the GC actually ran. 05:54
nwc10 good *, nine
06:25 robertle left 06:31 patrickb joined 06:48 domidumont joined 07:08 patrickb left 08:17 zakharyas left 08:27 zakharyas joined 08:36 sena_kun joined 08:39 zakharyas left
jnthn timotimo++ # ooh, will look forward to playing with that :) 08:49
09:32 brrt joined
brrt \o 09:32
nwc10 o/ 09:37
brrt is in the process of creating the dumbest ever List::Util replacement 09:51
nwc10: I kind of forgot how to do the exporter dance in perl5
nwc10 from memory, so might have errors
use parent 'exporter';
our @EXPORT_OK = qw(func func2 func3); 09:52
brrt thanks :-) 09:54
moritz or "use Exporter qw(import);" 09:57
brrt :-) 10:00
yeah, that's shorter still
ok
worlds worst List::Util reimplementation, comping up...
nwc10 you still need the @EXPORT_OK 10:01
and also you don't need to re-implement *all* of List::Util, just the bits not in 5.mumble
where 5.mumble is the minimum perl 5 version "we" feel like depending on
Anyone on anything earlier than 5.8.4 is likely doomed 10:02
brrt gist.github.com/bdw/2c89f23227fca6...a1348e43ad
aargh, it doesn't actually work :-( 10:04
the '$a' refers to the $a in the package...
this presumably works via the XS :-( 10:07
hmm, maybe it's simpler just to rewrite the code 10:10
10:28 domidumont left
Geth MoarVM: d79e7d5afc | (Bart Wiegmans)++ | tools/expr-template-compiler.pl
[JIT] Revert template compiler back to v5.10

List::Util subroutines pairgrep, pairkeys, and pairvalues aren't in perl 5.10 yet, so I reimpleneted the functionality in basic perl.		 10:31
brrt moritz++ for the find
dogbert17 brrt++ 10:46
10:46 brrt left 10:49 travis-ci joined
travis-ci MoarVM build passed. Bart Wiegmans '[JIT] Revert template compiler back to v5.10 10:49
travis-ci.org/MoarVM/MoarVM/builds/577277190 github.com/MoarVM/MoarVM/compare/1...9e7d5afc1a
10:49 travis-ci left 10:55 brrt joined
dogbert17 hi brrt, btw, timotimo uploaded new coverity scan results for MoarVM yesterday. There were some JIT related complaints although I guess they could be false positives 11:08
brrt yeah, I saw them 11:24
thanks
the missing break was a false positive
any idea how to silence that?
dogbert17 I believe that coverity supports some annotations that might help 11:32
community.synopsys.com/s/article/h...h-coverity 11:46
12:00 domidumont joined 12:21 patrickb joined
timotimo jnthn: should there also be a SpeshLog event that contains string data from the speshlog as an int8 buffer or so? 12:22
12:30 Kaiepi left
jnthn timotimo: Hmm....well, only if you opt in to that, I think. It'll be quite a slowdown to produce, but maybe very useful 12:30
12:30 Kaiepi joined
timotimo of course it's an opt-in thing 12:31
i'm thinking passing the data as int8 instead of Str will make it a little cheaper to ignore the stuff you're not interested in
though to be fair, our utf8 parsing isn't slow 12:32
jnthn I think spesh info by default just wants to be like start time, end time, file, line, was it jitted, inline count, and similar overview info
And getting the spesh log is an extra option
timotimo for sure
what else can you think of? 12:34
apart from spesh, i mean
like, should "loading bytecode" be an observable event?
12:45 lucasb joined
jnthn Perhaps yes. Also I did wonder about opening/closing of handles, so you can see if you forgot to close some file handle. 12:46
brrt observability, yay 12:51
timotimo if we have "loading bytecode", perhaps we should have a more precise "created compunit" vs "gc'd compunit" event so you'd be able to see when your program creates more and more code with eval that never gets freed 13:02
just spitballing here
damn it, now i remember than while i was falling asleep i seemed to recall some bug i had been working on at some point but probably forgot about 13:06
hm, it could have been the one where a task id for an asynctask gets used twice by two different task kinds 13:07
sena_kun timotimo, want a ssh? :) 13:12
timotimo oh, you were whomst had the bug?
sena_kun yeah
timotimo refresh my memory, what module or code did that involve? 13:13
sena_kun cro(cro-ldap), most of the time its ok, but once in a while it dies with non-AsyncTask (of type %s) fetched from eventloop active work list, and sometimes with just "incorrect number of arguments passed, wanted N but passed 7+/-" 13:14
timotimo OK, that helps 13:15
sena_kun we had an ssh session where you was able to record it with rr
s/you was/you were/ 13:16
timotimo it occurs to me that getting the gc start time from "start of VM" is helpful to compare across runs, but if, say, a whateverable bot were to put something like "last major GC was n seconds ago" in its uptime command, it'd have to associate the gc times with absolute time manually whenever it receives the notification on the queue or so
just pushing the absolute time first, and then the time-since-start-of-vm would be very cheap, though 13:18
also, i should be writing directly to the VMArray storage buffer rather than calling MVM_repr_push over and over, since the type's REPR ID is already forced to be VMArray anyway, and the slot type is also fixed 13:20
so going via virtual function calls is basically 100% waste
not much cost to speak of in total, but the lower the overhead, the better i reckon
brrt timotimo: JIT compile it? 13:27
:-P
timotimo that's basically what i'm doing, it's just that i'm doing it by hand :) 13:28
and ahead of time
but it's not like "ahead of time" is the opposite of "just in time" or anything
will also turn "MVM_repr_alloc_init" into "MVM_repr_alloc" since VMArray doesn't have an init function 13:30
brrt the opposite of 'just in time' is 'too late' 13:37
timotimo ha 13:39
dogbert17 todays stupid question, is there a break statement missing here? github.com/MoarVM/MoarVM/blob/mast...pea.c#L630 14:03
jnthn dogbert17: Yes 14:04
dogbert17 should I make a PR real quick? 14:05
dogbert17 does so unless someone beats me to it 14:06
jnthn Yes, go for it 14:07
Geth MoarVM: dogbert17++ created pull request #1165:
Add missing break statement		 14:16
14:18 sena_kun left
timotimo jnthn: how do you feel about a transformation for "object is created but never read from or written to, but deopt desires it" to "drop it and materialize it out of thin air during deopt"? 14:24
this still happens a bunch in situations where there's no named parameters passed to a speshed frame; oftentimes the named arguments hash gets created as an empty hash but never touched in the code 14:29
Geth MoarVM: 269b73ee04 | (Jan-Olof Hendig)++ | src/spesh/pea.c
Add missing break statement

Discovered by Coverity and confirmed as legit by jnthn++ Spectested without errors
MoarVM: 127d7839f6 | niner++ (committed using GitHub Web editor) | src/spesh/pea.c
Merge pull request #1165 from dogbert17/fix-missing-break

Add missing break statement
dogbert17 heh, is nine home from his vacation? 14:30
15:06 patrickb left 15:34 Kaiepi left, nativecallable6 left, committable6 left, evalable6 left, reportable6_ left, bisectable6 left, unicodable6_ left, notable6 left, releasable6 left, benchable6 left, greppable6 left, tellable6 left, squashable6 left, statisfiable6 left, coverable6 left, shareable6 left, quotable6 left, Kaiepi joined 15:35 squashable6 joined, statisfiable6 joined, bisectable6 joined, greppable6 joined 15:36 benchable6 joined, nativecallable6 joined, quotable6 joined, evalable6 joined, bloatable6 joined 15:37 releasable6 joined, tellable6 joined, committable6 joined 15:38 notable6 joined, unicodable6 joined, shareable6 joined 15:39 coverable6 joined, reportable6 joined 16:12 MasterDuke joined 16:23 robertle joined
brrt timotimo: no idea what jnthn said, but I think that transformation is a very good idea 16:23
timotimo he hasn't had the opportunity to react yet 16:25
jnthn Ah, missed it. But I think PEA already does it :)
brrt so, I'm totally unaware of stuff...
jnthn Or at least, the more-pea branch
brrt but I'd like a design where a deopt-check would go to a branch, rather than just drop out, and then a deopt-to operator would actually be the one that dropped to the interpreter 16:26
timotimo ah, great!
i should be playing around with more-pea
i agree with brrt, that'd give me the opportunity to more easily log deopt reasons for the profiler 16:27
brrt wonders whether to call the next branch garden-pea or chick-pea
timotimo deopt could be pushed to a subscription queue thingie fwiw
though the traffic there could be immense
worse if the code that reacts to the data deopts :D
brrt timotimo: limited buffer, with overflow = skip 16:31
?
(actually, drop)
timotimo ConcurrentBlockingQueue doesn't really support limited buffers, though?
maybe with the internal "get me the current approximate length of this queue" non-api it could work 16:32
16:51 pamplemousse joined 16:58 chloekek joined 17:04 brrt left 17:07 brrt joined 17:26 domidumont left
Kaiepi timotimo, i took a look at how async tcp sockets deal with the gc to see if async udp sockets really weren't marking the handle that keeps getting zeroed out since those actually work and... it's pretty well exactly the same, i don't see how that's happening 17:28
i didn't have to touch any code dealing with the gc
17:29 Ven`` joined
Kaiepi though i might have to, async udp writes could potentially take a lot longer than before 17:31
17:34 pamplemousse left
Kaiepi i'll probably need to make it possible to connect udp sockets once i properly start work on async networking to make up for it 17:37
maybe i'll pullreq what i have now as a wip to see if anyone can figure out why the handle's getting zeroed out like that 17:41
17:49 brrt left 18:27 chloekek left
timotimo makes the sql output a little less wasteful by using "insert into table values (a), (b), (c), ...;" syntax 18:42
instead of repeating the "insert into table values" part over and over 18:43
18:43 brrt joined
MasterDuke nice 18:45
timotimo: for your mvmeventsubscribe branch, does the new op need to be jitted? otherwise won't it give more pessimistic numbers (compared to running regular jitted, non-event-queue-substribing code) than it could? 18:51
timotimo it's supposed to be called only a few times during the life of a program 18:52
and definitely not in hot code :) 18:53
if someone is a dummy like me and puts it in their mainline code, they gotta get surprised, sure 18:59
but the op is supposed to disappear into an API that rakudo offers anyway
MasterDuke ah, ok 19:00
timotimo if it ends up causing jit bails, we can jit it 19:01
i mean we can jit it immediately
MasterDuke is there any coordination between spesh inlining decisions and jitting? is it needed? i.e., would it sometimes make sense to not inline something because an op inside it would cause the larger combined block to bail? 19:05
19:05 Kaiepi left
timotimo there is no coordination yet, and theoretically every op could be jitted :P 19:06
19:06 Kaiepi joined
brrt and ought to be 19:06
MasterDuke well, some haven't been yet because it's tricky, right? 19:07
timotimo there's a spooky newline that appers in the profiler output 19:10
ah 19:11
well, food time!
MasterDuke e.g, ctxouter, getcurhllsym 19:18
19:22 sena_kun joined 19:23 sena_kun left
Geth MoarVM: Kaiepi++ created pull request #1166:
[WIP] [IP6NS Grant] Improve hostname resolution		 19:30
MoarVM: rbnx++ created pull request #1167:
Improve MVM_bithacks_next_greater_pow2 function		 19:33
19:45 Ven`` left, Ven`` joined
Kaiepi waaaiiit a minute 19:53
i wonder if making MVM_io_async_socket_udp create the MVMOSHandle earlier than right before passing it to rakudo and mark it would fix the issue 19:57
20:09 brrt left
timotimo when i "alter table" inside of a transaction, all following inserts that leave out the value in question will get the "new" default, right? 20:17
Kaiepi ...nope
@ myself, not you 20:18
timotimo ah, ok
Kaiepi but trying it was enlightening because i forgot to null out the udp handle's data, which originally contained the SocketSetupInfo struct, allowing me to look at what the handle originally looked like 20:23
the MVMOSHandle's body's data is still null, even though i think i do assign it to what it's supposed to be...? 20:24
20:27 chloekek joined
timotimo ah, i can't "alter table" in sqlite to set a default value for a column 20:29
but maybe i can set the default value for a column to come from a select clause 20:35
Geth MoarVM: rbnx++ created pull request #1168:
change MVM_VECTOR growth factor from 2 to 1.5
timotimo whyyyy syntax error me :( 20:44
20:51 Ven`` left 21:02 pamplemousse joined
MasterDuke timotimo: i thought sqlite doesn't support alter table at all 21:08
timotimo yeah, it only allows adding columns or something 21:09
but i'm past that
right now it's not letting my insert statements through
want to take a look?
Kaiepi what are some other parts of moarvm besides src/io/asyncsocket.c and src/io/asyncsocketudp.c that deal with both async stuff with libuv and the gc? 21:10
timotimo gist.githubusercontent.com/timo/16...tfile1.txt
Kaiepi ah, src/io/procops.c does 21:12
MasterDuke timotimo: `INSERT INTO deallocations VALUES (337,3,1287,1,0,0), (408,3,1287,3,1,0));`, extra closing paren at the end, right?
timotimo oh
but the syntax error comes much earier 21:13
rebuilding ...
MasterDuke same with an `insert into gcs...` 21:16
timotimo i see a lone ); in a line of its own, too 21:17
MasterDuke and maybe types, but i can't keep the structure in my head with those json_objects()
timotimo i should get a big nqp program so i don't have to rebuild rakudo all the time 21:19
i think i got the last one now 21:23
wanna help me measure how much space is saved? :) 21:24
MasterDuke sure 21:25
timotimo huh 21:26
perl6 -ne 'state $saved = 0; state $prefixlen = 0; if /"INSERT INTO" .* "VALUES"/ { $prefixlen = $/.Str.chars }; if *.starts-with(" ") { $saved += $prefixlen }; END { say "saved { $saved / 1024 } bytes" };' < /tmp/shmup.sql 21:27
21:27 pamplemousse_ joined
timotimo i wonder why it spits out some "use of uninitialized * in numeric context" there 21:28
ah, that's why
perl6 -ne 'state $saved = 0; state $prefixlen = 0; if /"INSERT INTO" .* "VALUES"/ { $prefixlen = $/.Str.chars }; if .starts-with(" ") { $saved += $prefixlen }; END { say "saved { $saved / 1024 } bytes" };' < /tmp/shmup.sql
saved 841.169922 bytes
kbytes*
not bad at all 21:29
MasterDuke heh. don'thave a profile from before and after to compare size on disk?
but yeah, nice
timotimo can't really do that
i'd need a 100% predictable program 21:30
but i also would like multiple threads to be involved
next step will be to maybe go through the entire call graph twice, once for calls, once for allocations
that'll save a bunch more
MasterDuke the allocations/calls section has lots of interleaving, if any of that could be reduced it would help a lot
jinx
timotimo :)
i want to give the gcs table a default value for the thread number to be pulled from a different table so that i can just say "all gc entries from now on shall be for thread n" 21:31
that'll save like 3 to 4 bytes per gc run
MasterDuke save in the data structure the profiler builds in memory, or in the profile file written out? 21:34
timotimo profile file 21:35
the thread id is already not in the individual entries, it's known by where in the datastructure they live
MasterDuke ah 21:36
21:37 tobs left
timotimo 127422207 2.3M -rw-r--r--. 1 timo timo 2.3M Aug 27 23:44 /tmp/shmup.sql 21:44
saved 4491.818359 bytes
(kbytes)
ah, the value is incorrect 21:46
127422207 2.1M -rw-r--r--. 1 timo timo 2.1M Aug 27 23:49 /tmp/shmup.sql 21:49
saved 1424.177734 kibibytes
21:49 pamplemousse_ left
timotimo the sql is even wronger than beofre, great 21:50
MasterDuke heh 21:51
timotimo such long compilation times :| 21:53
21:55 chloekek left 21:56 tobs joined
timotimo OK, it saves almost 1.5 megabytes on a 2.1 megabytes sql file 22:05
that's super good
MasterDuke damn, that is good 22:08
timotimo 55kb on addition could be saved by making the extra lines indented with only one space, or 110kb if the indentation goes away completely
which is maybe what i'll do
welp, there's a PR now 22:26
MasterDuke any reason not to merge it? 22:28
timotimo not really i don't think 22:30
merged 22:31
MasterDuke cool
Geth MoarVM: vrurg++ created pull request #1169:
Separate includes from submodules from system-wide includes		 22:48
MoarVM: 3faf1985d2 | (Timo Paulssen)++ | 13 files
Introduce vmevent, "VM Event Subscription"

Lets a ConcBlockingQueue be configured to receive a set of events from inside the VM.
Currently there's only "gcevent", which gives you an object in the queue every time the GC runs. It's ... (6 more lines)		 22:54
MoarVM: 2bcb911d32 | (Vadim Belman)++ | 2 files
Separate includes from submodules from system-wide includes

Currently configuration variable cincludes contains `-I` flags for both includes from `3rdparty` for for those found with `pkg-config`. This is not correct as `cincludes` can be used by `NQP` and `Rakudo` for their C compilations. An example of a situation when this is necessary is rakudo/rakudo#3136 where `ffi.h` cannot be found even though `moar` is being compiled with it.		 23:01
synopsebot RAKUDO#3136 [open]: github.com/rakudo/rakudo/issues/3136 [BLOCKER][MoarVM][build] Rakudo 2019.07.1 fails to build on Gentoo Linux (no libffi-headers)
MoarVM: eb0cbca6ce | (Vadim Belman)++ | build/Makefile.in
Replace tabs with spaces for proper indentation
MoarVM: a37504d124 | (Aleks-Daniel Jakimenko-Aleksejev)++ (committed using GitHub Web editor) | 2 files
Merge pull request #1169 from vrurg/rakudo_3136

Separate includes from submodules from system-wide includes
Kaiepi ugh, i still can't tell what async udp sockets are doing wrong with the gc 23:07
timotimo i'd say show off your code, but at the same time i've gotta go to bed very soon 23:11
23:21 pamplemousse left
Kaiepi timotimo, i made a pullreq for it as is atm github.com/MoarVM/MoarVM/pull/1166 23:25
timotimo if i am to debug this, a one-liner that gets to the problem would be great, but i won't be able to do anything right now 23:27
Kaiepi perl6 -e 'my $s = IO::Socket::Async.bind-udp: "127.0.0.1", 8000; my $t = $s.Supply.tap({ .print }); my $c = IO::Socket::Async.udp; my $t2 = $c.Supply.tap({ .print }); await $c.print-to: "127.0.0.1", 8000' 23:35
that's what i've been using timotimo

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!