github.com/moarvm/moarvm | IRC logs at colabti.org/irclogger/irclogger_logs/moarvm
Set by AlexDaniel on 12 June 2018.
02:08 lizmat left 05:34 avar left 05:45 avar joined, avar left, avar joined, p6bannerbot sets mode: +v avar 05:46 p6bannerbot sets mode: +v avar 06:34 ilogger2 joined 06:35 p6bannerbot sets mode: +v ilogger2 07:03 domidumont joined 07:04 p6bannerbot sets mode: +v domidumont 07:48 robertle joined, p6bannerbot sets mode: +v robertle
jnthn timotimo: I forget exactly what it does; it may just let old ones be GC'd 09:30
Would have to check the code
timotimo that seems to be the case
i put a breakpoint on its gc_mark and for every gc run there was only ever one that got marked 09:31
so that shouldn't cause unlimited memory growth
jnthn Probably I did that because it's "obviously safe", whereas re-use always raises questions :)
Well, is its gc_free being called? :)
nwc10 jnthn: ASAN considers e317922036598184745c9b2afc1eafb44597e75b to be boring 09:36
(Sketch out creating deopt/materialization info)
jnthn Yay :) 09:44
nwc10 please try harder :-) 09:45
09:51 domidumont left, domidumont joined 09:52 p6bannerbot sets mode: +v domidumont
timotimo putting the breakpoint in the wrong spot will get you to a spot where you can "c" as often as you want and you won't make progress, because for some reason the other thread won't release the condition variable 10:01
ok, i've got some more breakpoints and i'm seeing lots and lots of spesh logs being created and the distance between nursery_alloc_limit and nursery_alloc only shrinking by a very small amount 10:14
so maybe we should deduct the size of the spesh log buffer from the nursery alloc limit 10:15
but even then, this shouldn't be unbounded growth 10:16
i'll wait for a gc run to happen
10:20 lizmat joined, p6bannerbot sets mode: +v lizmat 10:40 zakharyas joined 10:41 p6bannerbot sets mode: +v zakharyas
timotimo OK, it's not unbounded, the bound is just very, very, very high 10:46
RSS is currently at 490, the nursery still has 94992 bytes of space 10:47
the other thread also occasionally creates spesh logs, it's at 126472 bytes left before a GC would happen
i'd almost consider freeing the buffer right when the spesh worker is done with it, tbh 10:48
and letting the gc take care of the object itself, only freeing the buffer if it's not turned into a null pointer yet
m: say 93152 / 80 10:49
camelia 1164.4
timotimo m: say (1164 * 16384 * 24) / (1024 * 1024) 10:50
camelia 436.5
timotimo ok, it'd still allocate this many megabytes of ram before the GC kicks in
11:12 squashable6 joined 11:13 p6bannerbot sets mode: +v squashable6
Geth MoarVM/decrease_spesh_log_memory_growth: c933b51669 | (Timo Paulssen)++ | 2 files
free spesh log entries after consuming them

this prevents threads that do little to no allocations but lots of spesh logging in an otherwise idle program from growing to multiple gigabytes of ram used solely for spesh log entries.
This happened with rakudo's ThreadPoolScheduler supervisor thread when a program with multithreading or async workloads was left idle, like for example a Cro app.		 11:43
timotimo m: say (16384 * 24) / (1024 * 1024)
camelia 0.375
Geth MoarVM: timo++ created pull request #1002:
free spesh log entries after consuming them		 11:47
timotimo ^- pull request contains a bit more prose
anyway, that makes my program's memory growth stop 11:55
if it's still at 219 in an hour, i'd say it's done :) :) 11:56
nwc10 and you plan to test this by going out for lunch? 11:57
timotimo late breakfast, i believe 12:02
12:16 evalable6 joined, p6bannerbot sets mode: +v evalable6
timotimo AlexDaniel: wanna try out that branch to see if it makes things better? 12:19
AlexDaniel timotimo: I don't know any quick and easy way to test it 12:20
timotimo well, how slowly does it grow? 12:21
AlexDaniel well uhh… after like 2 weeks I think the memory usage grows to like two gigs of ram… or something like that
timotimo mhh 12:23
that's about what i expect from spesh log growth
timotimo AFK
12:31 shareable6 joined 12:32 reportable6 joined, p6bannerbot sets mode: +v shareable6 12:33 p6bannerbot sets mode: +v reportable6
AlexDaniel also just got a machine with 64 GB of RAM 12:45
maybe these memory issues don't matter as much after all :D
timotimo it'd still be nice to run on raspberry pis (or beaglebones) without blowing all the memory on unused data 13:00
oh no! 13:02
it grew to 200 megs! :(
13:19 avar joined, avar left, avar joined, p6bannerbot sets mode: +v avar
nine timotimo: but does it still grow as much? 14:02
timotimo it has now grown to 181 megs 14:17
or rather it has nworged to 181 megs
lizmat nworged > 14:42
?
[Coke] m: say "grown".flip ~ "ed" 14:44
camelia nworged
lizmat aha! 14:45
timotimo it's still at 181 megs right now 14:46
though i don't know if it went up some time in between
14:47 domidumont left 15:02 domidumont joined 15:03 p6bannerbot sets mode: +v domidumont 15:05 zakharyas left 17:30 domidumont left 17:36 robertle left
timotimo it nworged to 180 in the mean time 18:17
18:33 domidumont joined 18:34 p6bannerbot sets mode: +v domidumont
samcv timotimo, minor nitpick: can you make sure you use titlecase in your commit messages from now on? 19:10
timotimo oh 19:13
i bet there's a git commit hook on the internet that i could steal
samcv heh
timotimo title case means every word is capitalized, right? except for maybe words like "the" or "a"? 19:14
samcv luckily i have a (T)itlecase option in my update-changelog script
oh no just the first one
19:14 AlexDaniel` joined, p6bannerbot sets mode: +v AlexDaniel`
samcv nine, can you give me an overview of the reason for the MAST changes? 19:15
or someone else that knows
github.com/MoarVM/MoarVM/commit/4f...e51c0c8cd7
timotimo so far we had a set of classes like MAST::Label, MAST::Operand, ... and instruction lists that hold them 19:16
and that got fed as one big batch to mastcompiler.c
now the QAST is turned into mbc as directly as possible 19:17
this can greatly decrease the maximum memory used
samcv: do you think that PR can go into master before this release?
maybe i should at least stresstest it 19:18
samcv link to PR
since what i linked is already merged, so guessing you mean a differentn one 19:19
19:20 zakharyas joined
timotimo github.com/MoarVM/MoarVM/pull/1002 19:20
19:21 p6bannerbot sets mode: +v zakharyas
samcv timotimo, i'm going to merge. let me know how the stresstest goes 19:22
Geth MoarVM: c933b51669 | (Timo Paulssen)++ | 2 files
free spesh log entries after consuming them

this prevents threads that do little to no allocations but lots of spesh logging in an otherwise idle program from growing to multiple gigabytes of ram used solely for spesh log entries.
This happened with rakudo's ThreadPoolScheduler supervisor thread when a program with multithreading or async workloads was left idle, like for example a Cro app.		 19:23
MoarVM: 88a776d1ab | (Samantha McVey)++ (committed using GitHub Web editor) | 2 files
Merge pull request #1002 from MoarVM/decrease_spesh_log_memory_growth

free spesh log entries after consuming them
timotimo i haven't updated rakudo recently, so a few individual tests fail, but the only symptom that could come from this is terrible memory corruption or very bad mishebaviour 19:26
maybe nwc10 can run it with asan? :)
jnthn We can perhaps think about removing src/mast/ stuff, given I guess we're not using it now we generate the bytecode directly 19:28
(After the release, of course)
timotimo i'm glad i got around to figuring out the memory growth thing 19:29
i've gotta go grocery shopping, but maybe i can ssh in later and report on the stresstest 19:30
jnthn timotimo++ 19:31
Geth MoarVM: 856c4dd880 | (Samantha McVey)++ | 2 files
Make MVM_string_utf16*_decode consistent and use char*

Fixes a bunch of type warnings.		 19:33
MoarVM: 4228ad1b07 | (Samantha McVey)++ | tools/update-changelog.p6
Use ANSI bold to make update-changelog.p6 easier to read

Cosmetic changes.
19:40 ilmari_ joined, wictory[m] joined, p6bannerbot sets mode: +v wictory[m], p6bannerbot sets mode: +v ilmari_ 19:50 patrickb joined 19:51 p6bannerbot sets mode: +v patrickb 19:52 domidumont left 19:56 ilmari_ left, AlexDaniel` left, wictory[m] left 19:59 travis-ci joined, p6bannerbot sets mode: +v travis-ci
travis-ci MoarVM build failed. Samantha McVey 'Use ANSI bold to make update-changelog.p6 easier to read 19:59
travis-ci.org/MoarVM/MoarVM/builds/460426305 github.com/MoarVM/MoarVM/compare/8...28ad1b0798
19:59 travis-ci left 20:05 AlexDaniel` joined, p6bannerbot sets mode: +v AlexDaniel`
dogbert11 hmm, a sudden SEGV 20:11
==19776==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f73a87bc830 bp 0x7fff237b19b0 sp 0x7fff237b19b0 T0) 20:13
#0 0x7f73a87bc82f in has_little_endian_bom src/strings/utf16.c:9
#1 0x7f73a87bc82f in MVM_string_utf16_decode src/strings/utf16.c:207
samcv i made a typo in my change i think :) 20:14
jnthn
.oO( Somebody set up us the bom )
dogbert11 pesky typos :) 20:15
Geth MoarVM: c9f4863170 | (Samantha McVey)++ | src/strings/utf16.c
Fix typo causing segfault in utf16 code

Fixes a typo in 856c4dd880b32514c033371d46d0951398f5c2f0		 20:17
samcv dogbert11, should be fixed now
dogbert11 cool 20:18
dogbert11 rebuilds
samcv jnthn, not sure where the best place to put arch conditional ldflags in Configure.pl 20:20
dogbert11 works perfectly, samcv++
samcv and make sure it works for cross as well
jnthn samcv: I don't know; I'm pretty weak at build/configure stuff, so most of what we have today has been done by folks who understand that better :) 20:21
samcv: This is for the AIX fix? 20:22
samcv yeah 20:30
something about memory allocation that was reported to me a while ago. that i remembered recently :) 20:31
timotimo samcv: stresstest was OK :)
samcv github.com/MoarVM/MoarVM/issues/1003
timotimo, nice!
20:32 wictory[m] joined, ilmari_ joined, p6bannerbot sets mode: +v wictory[m], p6bannerbot sets mode: +v ilmari_, lizmat left 20:36 lizmat joined, p6bannerbot sets mode: +v lizmat 20:39 travis-ci joined, p6bannerbot sets mode: +v travis-ci
travis-ci MoarVM build passed. Samantha McVey 'Fix typo causing segfault in utf16 code 20:39
travis-ci.org/MoarVM/MoarVM/builds/460446200 github.com/MoarVM/MoarVM/compare/4...f486317072
20:39 travis-ci left 20:43 zakharyas left
Geth MoarVM: 60c16e6cbe | (Samantha McVey)++ | docs/ChangeLog
Update ChangeLog for release
 21:03
samcv ok. changelog is pushed. timotimo can you look at the "Other" section and make sure my MAST description is alright? 21:04
timotimo samcv: i wouldn't say "MAST class changes", because MAST was pretty much completely removed 21:05
a detail i should probably have mentioned is that what used to be in mastcompiler.c is now implemented in nqp code
i gotta go for now, though. food ain't gonna cook itself! 21:06
samcv kk 21:08
timotimo "expose first entry time in profiler data" is in theret wice 21:34
samcv oops
timotimo otherwise i didn't see anything wrong 21:36
samcv but say that MAST was mostly removed? and is now implemented in nqp code?
so why does this consume less memory if it's using nqp and not c code? is that just because it's better for other reasons? 21:37
timotimo 21:50
22:16 patrickb left
timotimo yeah, it's better for other reasons 23:15
sorry for answering so late
good foods was had
samcv: still around? i should be able to explain 23:16
export EXPLAIN_FRONTEND=noninteractive 23:56
so the previous way required all of the QAST to be transformed into a whole bunch of MAST::Blah objects, and there would be one object for each operation and one for each register, or lexical, or label, ... 23:58
and since there's only a single op that takes all the MAST objects and turns them into a single mbc compunit, the entire MASTified QAST tree for the whole comp unit has to be in memory at that one point in time, along with the original QAST, because that doesn't get collected just yet
(that single op is masttocu, i believe, which is how mastcompiler.c's code has been invoked so far) 23:59

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!