timotimo i had suspected it in line_coverage.c 00:00
wonder if the output files would get trashed by multiple threads running code simultaneously
i could actually re-use telemetry for this and make coverage logging and telemetry mutually exclusive 00:03
MasterDuke seems reasonable 00:04
timotimo the data delivery mechanism of telemeh only permits a little bit of data for every "packet" 00:05
MasterDuke enough for filepath + linenum? 00:07
timotimo nope 00:11
the biggest is long long + int + const char * 00:12
and also the thread id
MasterDuke hm, create some sort of hash of filepaths and just pass keys/indexes? 00:13
timotimo hmm, are the strings in the annotation perhaps null terminated ...
MasterDuke afk for a bit 00:14
timotimo hm, if i serialize instrumentation barriers so that i am guaranteed exclusive access to a datastructure to register loc info IDs ... 00:27
ooh, /me learns about the quota boost 00:49
jnthn sleep & 01:01
MasterDuke i feel like there's some joke to be made about upgrading my Perl 6 interpreter with nitrous 01:04
timotimo: fyi, `perl6 -e 'my $a; for ^100_000 { $a = $_.comb.sort.unique }; say $a'` took 4s normally, 8s with MVM_COVERAGE_LOG=c0.log and MVM_COVERAGE_CONTROL=2 01:11
interestingly, when i misspelled .unique as .uniq, with those env variables set i got `MoarVM oops: Spesh: failed to fix up inline 1` instead of `No such method 'uniq' for invocant of type 'Seq'. Did you mean 'unique'? in block <unit> at -e line 1` 01:13
01:52 ilbot3 joined 04:33 channing1 joined 04:41 channing1 left 06:24 buggable joined 07:50 Geth joined 08:08 Ven joined
Geth MoarVM: 074145ce5f | (Jan-Olof Hendig)++ | src/io/asyncsocket.c
Remove unused variable

No longer used after commit 2b0739d8		 08:10
MoarVM: 1be02e0363 | (Jan-Olof Hendig)++ | src/core/fixedsizealloc.c
Remove another unused variable

Possible copy/paste error
MoarVM: 715447df4c | niner++ (committed using GitHub Web editor) | 2 files
Merge pull request #628 from dogbert17/remove-unused-vars

Remove unused vars
09:30 dogbert11 joined 09:58 Ven joined 10:08 AlexDaniel joined 10:42 Ven_ joined
dogbert11 just tried to build the 'even-moar-jit' branch but the configuration step failed with: 11:46
Generating src/gen/config.h ............................ FAIL
unknown configuration key 'jit_arch'
what could be the reson for this? 11:47
is it because I'm on 32-bit?
11:59 Ven joined
MasterDuke dogbert11: hm, maybe Configure.pl around line 403 needs another branch in that if statement 11:59
dogbert11 MasterDuke: I was tricked, it did complain with the message 'JIT isn't supported on platforms with 4 byte pointers.'. However this message is hidden in the config output which leads me to wonder why the config process isn't stopped at that point. 12:31
MasterDuke yeah, that's odd 12:32
dogbert11 it just prints the message and continues only to fail completely a short time later with the 'jit_arch' message 12:33
MasterDuke hm, so we lose 32bit support completely with that branch? or is there a way to tell it to not build the jit? 12:35
dogbert11 I don't believe that the jit has ever worked on 32 bit for some unspecified reason 12:36
jnthn I suspect the configure problem is just an accident rather than anything deep 12:37
MasterDuke yeah, but you can build the current moar without it
jnthn From the error, it seems like it's trying to find a jit-related bit of configuration and include it in a non-JIT build
12:37 http_GK1wmSU joined
jnthn dogbert11: Reason is that x64 machine code and x86 machine code are quite different things to produce. :) 12:38
dogbert11 ah, simple as that :) 12:39
jnthn And so producing the x86 code as well is a substantial amount of work, though the expr JIT will make it possible to do so with less effort
MasterDuke fyi, on a semi-related note, i just built the even-moar-jit branch, built nqp, passed nqp's tests, built rakudo, and then passed its test and spectest 12:40
Geth MoarVM: f3fc2029db | MasterDuke17++ | src/strings/ascii.c
Del unused var in MVM_string_ascii_encode_substr

The variable was just a cast of an argument and only used once, just cast the argument directly.
MoarVM: a9c421b1a3 | MasterDuke17++ | src/strings/latin1.c
Del unused var in MVM_string_latin1_encode_substr

The variable was just a cast of an argument and only used once, just cast the argument directly.
MoarVM: cbb1faa1bb | (Jonathan Worthington)++ (committed using GitHub Web editor) | 2 files
Merge pull request #619 from MasterDuke17/remove_unnecessary_variable_in_MVM_string__encode_substr

Remove unnecessary variable in MVM_string_(ascii|latin1)_encode_substr
dogbert11 should the new jit branch lead to any performance increases?
12:40 http_GK1wmSU left
Geth MoarVM: 123dbdb929 | (David Warring)++ | src/6model/serialization.c
attempt at a better error message for RT#126341
 12:41
MoarVM: 8752f9b0fe | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/6model/serialization.c
Merge pull request #617 from dwarring/rt126341

attempt at a better error message for RT#126341
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=126341
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=126341
MasterDuke my spectest time was roughly in line with before, but i don't have exact numbers to compare
dogbert11 jnthn++ new blog post 12:42
could it be said that Perl6 have three different optimizers, i.e. the jit, spesh and the --optimize switch in the perl6 executable? 12:43
timotimo hm, i suppose so 12:51
though the current jit doesn't optimize much
the new jit is an optimizer compared to the current jit :P
but will get more stuff as time passes
dogbert11 very cool, I guess I'll have to switch to a 64 bit vm sooner or later 12:52
timotimo but yeah, the --optimize flag in the perl6 executable is completely distinct
i'd recommend that :)
dogbert11 how does that work, i.e. what kind of optimizations does it do?
timotimo it's written in nqp and operates on the QAST representation of the code 12:53
you can get a look at what it does by comparing --target=ast and --target=optimize
dogbert11 so that optimizer works on a higher language level than both spesh and the jit? 12:54
timotimo i'd say the most important thing it does is turn lexical variables into local variables because only blocks without lexical variables can be directly inlined into their lexical outers
that's right, but it also doesn't have any run-time information
dogbert11 so it does its job at program startup/parsing? 12:55
timotimo during compilation, yes, including precompilation of modules
dogbert11 interesting indeed 12:56
timotimo another thing is that the MAST generated from the QAST is sometimes quite ugly 12:57
spesh can help there, too
like an abundance of decont and hllize instructions that don't actually do anything in 99.9% of code execution 12:58
dogbert11 hmm, slightly confusing terminology though, ASA, QAST, MAST and CFG's in SSA form
ASA -> AST 12:59
timotimo well, QAST is just "what comes after PAST" and PAST was Parrot AST 13:10
13:10 lizmat joined
timotimo MAST is a moarvm-specific format that's no longer tree-shaped and very close to moarvm bytecode 13:11
dogbert11 thx for the explanation timotimo 13:23
timotimo no prob 13:25
i was going to point you at the mastcompiler c file in moarvm and say "that's why this file is so tiny and simple" 13:26
but it's still pretty big :P
13:27 AlexDaniel joined
timotimo moar is doing something strange 14:03
opening /proc/self/stat multiple times per second
checking if it should do a full gc, huh 14:06
can't be too slow i suppose; it doesn't show up notably in the perf output 14:07
15:12 zakharyas joined
AlexDaniel MasterDuke: hey, so what's the current status of the leak? 15:42
15:45 pharv joined
timotimo hm, so, heaptrack, right? can we get it to show more than just the caller of the allocator function itself? 15:46
16:08 Ven joined
MasterDuke timotimo: would the flame graph be more useful? i.imgur.com/dJXfLVF.png 16:51
timotimo yeah that looks interesting 16:52
does "allocations" mean "how often was any allocator function called"?
is there one that tallies up the size? 16:53
MasterDuke i.imgur.com/MobPJRy.png
timotimo ah dang
MasterDuke ? 16:54
timotimo whatever function ends up triggering the gc will get the nursery re-allocation counted towards it
16:54 Ven_ joined
timotimo that won't do at all 16:55
however, we do see that the "little" buffers we allocate for libuv to fill make up a big chunk of what we allocate in total
MasterDuke i.imgur.com/J20JZv5.png by peak allocation 16:56
do you have heaptrack installed?
timotimo i do 16:58
i think?
hm, okay, the peak of stuff allocated there isn't terribly bad
MasterDuke i could put my heaptrack recording on the whateverable server if you want to play around with viewing it locally 16:59
timotimo interesting 17:03
hm but now
this is leaking, right?
i.e. increasing its memory
MasterDuke yeah 17:04
timotimo does any part of the flame graph get noticably bigger or does it all just sort of grow together?
MasterDuke i.imgur.com/CnqmYw8.png
timotimo: don't quite understand the question 17:08
timotimo like do all parts grow equally when you increase the number of iterations? 17:16
17:29 Voldenet joined
timotimo can you hide individual slices from that mountain graph? 17:35
everything above the orange is really difficult to make out because it just goes up and down so much 17:36
MasterDuke you can zoom in and hover over the sections and it'll tell you the function they're for. hard to show that in a screenshot though 17:43
timotimo right
might have to actually use the heap snapshot profile for this task
MasterDuke but it was causing immediate segvs 17:44
timotimo wow, damn 17:45
MasterDuke fyi, i just put the heaptrack recording on the whateverable server
if you want it
timotimo interesting, it segfaults inside spesh 17:46
without spesh the heap profiler can run
jnthn Huh, that's a bit odd...
timotimo when gc-ing sim frames 17:47
MasterDuke maybe related, but did you notice my comments from late last night ?
interestingly, when i misspelled .unique as .uniq, with those env variables set i got `MoarVM oops: Spesh: failed to fix up inline 1` instead of `No such method 'uniq' for invocant of type 'Seq'. Did you mean 'unique'? in block <unit> at -e line 1`
timotimo i only looked at it a microscopical bit
so don't know what actually is broken here
jnthn The spesh worker thread runs just as a normal thread, so...
MasterDuke: Can you file that as a MoarVM issue? I'd like to look at that during the week.
MasterDuke jnthn: the heap profiler thing? or the "failed to fix up inline"? 17:48
jnthn MasterDuke: The inline one 17:49
MasterDuke k
jnthn I changed a bunch of inline-related things last week so we can actually deleted dead inlines.
*delete
It should mark what it removes
And then not try to fix them up
And I remembering writing the code to make it handle nested inlines 17:50
So not sure what'd be going on. Will look in the next days.
MasterDuke github.com/MoarVM/MoarVM/issues/629 17:52
jnthn Thanks!
jnthn figures he should try and rest some ahead of throwing himself into another week of MoarVM hackery :) 17:53
MasterDuke also, jnthn++ for spesh blog post. very understandable 17:54
jnthn Glad it made some sense; always have to be careful writing such things after having been dug into the problem space for weeks :) 17:59
dogbert11 MasterDuke: I believe I misunderstood your memory problems yesterday, the problem is not memory leaks but rather memory usage. Is that correct? 18:07
18:08 lizmat joined
MasterDuke i'd say both 18:09
dogbert11 running your gist from yesterday (with four iterations rather than 20), leaks are negligible but allocated memory is soething else ... 18:10
MoarVM version 2017.07-333-g8752f9b0: total heap usage: 748,062 allocs, 433,024 frees, 1,337,211,640 bytes allocated 18:11
MoarVM version 2017.07: total heap usage: 643,447 allocs, 374,708 frees, 417,122,198 bytes allocated 18:12
MasterDuke yeah, i think there's some of both, but the large allocations is more noticeable
hm, wonder if i should do some manual bisecting with valgrind on the whateverable server 18:14
AlexDaniel bisecting bisectable again? :) 18:23
MasterDuke turtles all the way down! 18:25
dogbert11 hmm, the gisted code breaks --profile 18:28
dogbert11 tries another program 18:30
japhb
.oO( "Bisecting the bisectables but they don't know / Debugging the buggables but in the end you know / You turn away, what can I say?" )
MasterDuke dogbert11: yeah, pretty much anything with run/Proc::Async/etc breaks profiling 18:31
18:39 Ven joined 19:06 Voldenet_ joined 19:11 leego joined
MasterDuke looks like it's github.com/rakudo/rakudo/commit/0564891 19:13
valgrind says 2,037,358,560 bytes allocated after that commit, 378,997,214 bytes allocated before that commit 19:15
for 5 iterations in my test script
19:18 ggoebel joined, timotimo joined, tbrowder joined
AlexDaniel :o 19:24
19:47 brrt joined
brrt .tell jnthn: yeah, i need to rethink how i do the jit-compiler-disabling 19:49
yoleaux brrt: What kind of a name is "jnthn:"?!
brrt .tell jnthn i need to rethink how to disable the jt compiler at build time, now that it is so much larger than before
yoleaux brrt: I'll pass your message to jnthn.
dogbert11 MasterDuke++ 19:51
brrt: irclog.perlgeek.de/moarvm/2017-08-06#i_14974983
brrt i saw it. travis ci also saw it.
:-)
dogbert11 yay 19:52
brrt root issue: we're trying to compile a bunch of stuff that doesn't need to be compiled
i'm wondering how i can make it not do that 19:53
well, i know a way..
dogbert11 sounds promising
brrt it sounds like it would be a hack; i could wrap the entire thing with an ifdef 19:54
which would work, but i'd rather not
has to be a better way 19:55
anyway, i'm off again :-)
20:32 zakharyas joined
MasterDuke looks like it's github.com/MoarVM/MoarVM/commit/5c...46eb9b9058 20:47
timotimo but that should only cause churn, i.e. things being malloced and soon freed again 20:48
MasterDuke well this is just based on valgrind's "total heap usage" line 20:51
the last "allocated" number
before that commit it's ~300,00,00, after it's ~2,000,000,000
timotimo that's "total allocated" but not "in use at end" right? 20:52
MasterDuke right, that's number is pretty much the same 20:53
timotimo yeah, so it's just useless information :)
MasterDuke hm. what would be more useful? 20:55
timotimo hm
figuring out what kind of object there is more of after each iteration
but if you want you can probably revert that commit to get a less inflated number there 20:56
MasterDuke think it'll revert cleanly?
how would i get a count of objects if i can't use the heap profiler? 20:59
timotimo does it also crash if you turn spesh off? 21:01
MasterDuke not immediately... 21:03
ah, got a snapshot with spesh disabled 21:08
let see, guess i should get one at 2017.07 to compare 21:09
ugh, 2017.07 gave `MoarVM panic: Memory allocation failed; could not allocate 72085436 bytes` 21:13
MasterDuke needs moar ram!
21:16 ilmari[m] joined 21:20 zakharyas joined
MasterDuke had to reduce the number of iterations in my script to 3 21:29
but now i have a heap profile, with 43 snapshots 21:30
which one(s) should i look at?
timotimo oof 21:36
we don't really have a way to cross-reference what happens in the program with the heap snapshots, except for looking really closely 21:37
one of the reasons i'd like to offer an op that lets you put a piece of data into telemeh
MasterDuke would one at random in the middle be somewhat representative? 21:40
timotimo i suppose, but i'd rather go near the end
MasterDuke hm, 43 snapshots at 2017.07, only 34 at head 21:41
timotimo mhm 21:42
i forgot what exactly it was that jnthn did to often cause us fewer gc runs
MasterDuke compare 3/4ths of the way through of each?
timotimo mhm
MasterDuke say #35 and #28?
ugh, and don't have to ram to open both at once... 21:43
timotimo oops :( 21:44
on my laptop i have as much zram as i have actual ram
ever since i destroyed the slotted-in ram (or whatever else) i'm suffering under low ram all the time
MasterDuke i don't have a swap be default
timotimo oh
i have 2x zram compared to normal ram
it's actually already as full as the ram totally (i.e. including the compressed parts)
MasterDuke top objects by size? anything else useful? 21:46
MasterDuke doesn't have enough ram to have two heap analyzers open on any one computer, but just realized he's using two computers... 21:49
timotimo maybe also top objects by count and top stables by size and count 21:53
and maybe even top frames by size
MasterDuke 2017.07 gist.github.com/MasterDuke17/2cad4...96688f031f 21:55
HEAD gist.github.com/MasterDuke17/f0220...6a8948cc42 21:57
21:59 pharv joined
timotimo that's a whole lotta buf 22:00
MasterDuke but more at 2017.07 than HEAD 22:04
timotimo huh 22:05
22:13 Voldenet joined 22:53 MasterDuke joined 23:03 Ven joined 23:19 pharv joined
MasterDuke jnthn: does any of the the above point anything out to you? something else timotimo and/or i should try? 23:24
jnthn MasterDuke: Not really; I'll see if I can figure it out a bit more tomorrow though.
Heading to sleep shortly
MasterDuke cool, later... 23:26
timotimo i feel like sleep, too 23:31
have you tried the realloc trick for the bufs that we allocate for libuv btw?
MasterDuke no, didn't figure out what exactly was getting realloced to what value 23:34
but can try if you can give some more detailed instructions
timotimo oh, sure 23:36
what file was that in again ...
MasterDuke procops.c, somewhere around line 531 i believe
timotimo right, so buf->base is the pointer that's eligible for reallocing, buf->len is the actual size that it was last allocated with, and nread is how much is actually used inside the buffer 23:38
since we don't expect the buffer to be changed afterwards, we can probably just realloc it to the exact number of items read
MasterDuke jnthn had some comment about how buf->len is supposed to be correct (or something like that)
timotimo if we decide to, for example, round up to the next number aligned to, like, 32 bytes, or whatever, we'd have a different ssize vs elems 23:39
MasterDuke irclog.perlgeek.de/moarvm/2017-08-05#i_14973403 23:40
so i shouldn't just change what res_buf->body.ssize gets assigned, i need to do a realloc? 23:41
timotimo ssize will hold what you realloc to, which may be different from the number of elements used 23:42
MasterDuke so realloc, then set ssize to whatever value i used for the realloc? 23:43
btw, i assume the new value should be nread*(sizeof an element), but what is that size? 23:44
oh, tc->instance->str_consts.buf_type perhaps? 23:47
timotimo no, that's all just strings
we use 8 bits for all of these 23:48
that's also why the .i8 slot is selected for assignment 23:49
(even though it makes no difference)
(all platforms we develop for have the same size of pointer for all kinds of things pointed at)
MasterDuke ah, so just nread*1 ?
timotimo i believe so 23:52
MasterDuke `error: assignment of member ‘base’ in read-only object` 23:55
1519408maxresident)k before, 1303076maxresident)k after 23:59
timotimo um, huh?
MasterDuke res_buf->body.slots.i8 = MVM_realloc(buf->base, nread); res_buf->body.ssize = nread;
timotimo oh, huh

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!