Geth_ nqp: 55f9f85617 | Coke++ | docs/ops.markdown
fix formatting

  ... pass another doc test		 00:07
[Coke] looks like "force_gc" should be renamed to "forcegc" to match underscore usage. 00:10
Geth_ nqp: ba0bab4ebb | Coke++ | docs/ops.markdown
document force_gc
 00:14
00:14 softmoth_ joined 00:17 softmoth left
Geth_ nqp: d8b43c3110 | Coke++ | docs/ops.markdown
perl6 -> raku
 00:18
00:20 softmoth_ left, softmoth joined 00:26 softmoth left 00:36 softmoth joined 00:45 softmoth left 00:46 hungrydonkey joined 01:07 cognomin_ joined 01:10 cognominal left 01:19 cognominal joined 01:22 cognomin_ left 01:31 cognomin_ joined 01:35 cognominal left 02:01 hungrydonkey left, hungrydonkey joined 02:08 hungrydonkey left 02:09 hungrydonkey joined 02:11 hungrydonkey left 04:02 MasterDuke left 04:32 samebchase-2 joined 04:34 sjn_ joined 04:37 Kaeipi left 04:40 sjn left 04:46 tailgate joined 05:03 Kaiepi joined 06:42 [TuxCM] left 06:44 MasterDuke joined
MasterDuke timotimo: where should i stick a call to MVM_profiler_measure_gc_entry_size? 07:04
07:27 moon-child left
tyil .tell jmerelo GSoD applications have opened 07:30
tellable6 tyil, I'll pass your message to jmerelo
07:34 moon-child joined 08:21 Altai-man_ joined
lizmat Files=1306, Tests=111232, 213 wallclock secs (28.62 usr 7.80 sys + 2992.68 cusr 270.02 csys = 3299.12 CPU) 08:22
08:39 unclechu left, AlexDaniel` left 08:50 AlexDaniel` joined 09:11 unclechu joined
timotimo MasterDuke: in your gdb console :) 09:52
MasterDuke timotimo: oh, anytime? 09:56
timotimo yeah
well, profiling should already have started
MasterDuke have a couple printouts so far 10:01
10:05 sena_kun joined 10:06 Altai-man_ left
MasterDuke timotimo: gist.github.com/MasterDuke17/f0f3f...610bd6ef7f the compile is still going, but here's what i did so far 10:13
timotimo yeah, that call graph is hella deep 10:14
each of the numbers in the big block is one depth layer 10:17
i crunched some data with a friend yesterday, and we found that a majority of nodes only have few of their counters with any values 10:18
Geth_ rakudo: 2d1ba433af | (Elizabeth Mattijsen)++ | src/core.c/Num.pm6
Make Num.Rat conversion again 1.5x as fast

  - reduce Scalar allocations as much as possible by binding
  - use nqp::ops as much as possible to prevent unneeded coercions
Again, should affect performance of `now` similarly.		 10:20
timotimo that won't necessarily give us a 2x reduction 10:21
MasterDuke timotimo: so some space reduction possible? 10:22
timotimo but at this point i guess any difference is appreciated
MasterDuke btw, need any more printouts?
gist updated with a printout in the later stages 10:31
[Tux] Rakudo version 2020.02.1-313-g2d1ba433a - MoarVM version 2020.02.1-80-g32dfe7507
csv-ip5xs0.704 - 0.724
csv-ip5xs-206.279 - 6.688
csv-parser24.189 - 24.461
csv-test-xs-200.384 - 0.384
test7.492 - 7.663
test-t1.942 - 2.001
test-t --race0.823 - 0.985
test-t-2031.656 - 33.305
test-t-20 --race9.701 - 10.010
 10:40
10:43 committable6 left
Geth_ rakudo: 43c7e96f9a | (Elizabeth Mattijsen)++ | src/core.c/Num.pm6
Make FatRat handling use positionals

No need to use a named parameter, just pass on the type you want to have the object created in. Saves 1 scalar allocation per call
  (which means 17 instead of 18 garbage collections in $a.Rat for ^50000)		 10:48
10:49 committable6 joined 10:53 committable6 left 10:54 committable6 joined
Kaiepi m: CATCH { default { say 'ok' } }; @(supply { take 1 }).elems; say 'oh no' 10:56
camelia oh no
Kaiepi so this is a thing 10:57
but any fix i can come up with has a significant impact on its performance in a pretty bad way
timotimo we can also keep a list of types and staticframes and turn the pointers in the nodes into indices
Kaiepi maybe with the refactors to stuff like ... that'll change though
10:58 committable6 left
lizmat Kaiepi: not sure how ... refactor would affect that ? 10:58
jnthn m: dd @(supply { take 1 }).elems 10:59
camelia 1
jnthn m: dd @(supply { take 1 })
camelia (1,)
Kaiepi this happens because the gather block in Supply.list is a little too eager with what code belongs inside it
jnthn Oh hah, 'cus there's a `gather` somewhere on the call stack?
Kaiepi yep
jnthn It's just 'cus of dynamic scoping I guess 11:00
Kaiepi iirc your changes were going to affect stuff like gather lizmat?
11:00 committable6 joined
timotimo m: dd @(supply { take 1; break; take 2 }).elems 11:01
camelia 5===SORRY!5=== Error while compiling <tmp>
Undeclared routine:
break used at line 1. Did you mean 'last'?
lizmat no, I'm not touching gather
timotimo m: dd @(supply { take 1; last; take 2 }).elems
camelia 1
Kaiepi oh
lizmat Kaiepi: I'm re-implementing ... in terms of Iterator and not use gather / take at all
Kaiepi hm, i wonder what makes it so much slower when its done correctly then 11:02
lizmat take is a control exception
?
Kaiepi i tried moving the code that causes that outside the gather and giving Supply an iterator to work with, but both made @(supply { emit 1 }) somewhere around 1.5x-2x slower 11:03
but maybe the tradeoff isn't worth worrying about if the code's incorrect
lizmat afaik emit is also a control exception ?
m: CONTROL { dd .^name }; emit 42 11:04
camelia "CX::Emit"
emit without supply or react
in block <unit> at <tmp> line 1
Kaiepi i think so
still, i'm not sure why tapping the block outside the gather affects things to that degree 11:05
is it a scope thing? 11:06
lizmat could well be
Kaiepi could be possible to work around then 11:07
timotimo MasterDuke: i'll add a little bit of code to count nodes exactly, so we can calculate wins from stuff like "turn pointers into array indices" 11:20
oh, no way! 11:27
MasterDuke timotimo: you can't just leave us at that 11:49
timotimo in log.c do you see the line that increases alloc_alloc by 8?
Kaiepi ah hah! the problem was indeed scoping 11:50
MasterDuke line 362?
timotimo i think so, yes 11:51
turn that into += 2
MasterDuke profiles will use less memory? 11:56
timotimo see for yourself
the version of the debughelper branch i just pushed will show you what's what, too 11:59
in this regard
though you may want to revert the growth strategy change that's also in there
just to appreciate the difference :)
nine: you may not have to upgrade to a machine with bigger ram just yet; the counters for allocated objects were extremely wasteful 12:01
nine: to the point where one random measurement gave me a total size of 550 megs, 208 megs of which were allocated for alloc counting, 40.5 meg of which were actually used 12:02
MasterDuke += 2 or *= 2?
timotimo *= 2 will not work, since it starts out at 1
since it starts out at 0
that's why the patch i pushed uses ++ if it's at 0 before doubling
nine: in other words, 4/5ths of 2/5ths of the entire data was wasted :) 12:03
MasterDuke i changed it to += 2 and currently res is at 8.6gb, we'll see how high it gets
timotimo did you grab the new version of the measurement function?
MasterDuke not yet
12:03 Altai-man_ joined
MasterDuke i also rebased my add_new_ops_for_alloc_profiling branch on top of yours, so i'm collecting more data than you probably are 12:04
timotimo ah, yes 12:05
12:06 sena_kun left
MasterDuke finished stage parse, now at 10gb. don't remember exactly how high it got before, but at least 12gb 12:08
lizmat looks in awe
timotimo at the cost of making types and static frames live longer than they should, the garbage collector wouldn't have to go through the entire call graph any more, either 12:09
though it should do so every couple GC runs to make sure things are proper 12:10
Kaiepi if my changes pass tests, Supply.list will become around 20% faster
the way i dealt with the scoping issues might be useful for most of the other iterators in rakudo as well 12:11
timotimo that will make entering new call graph nodes and encountering new allocations slower, though, which is not fantastic 12:12
Kaiepi++
MasterDuke finished optimize, 10.3gb 12:13
timotimo oh, that's surprisingly little growth for that 12:15
MasterDuke process finished, but no profile written (i missed seeing if mem grew any higher) 12:24
timotimo d'oh 12:29
does profile-compile perhaps never write profiles actually? 12:30
MasterDuke it does on smaller jobs. it's the same bug 12:31
lizmat raku --profile-compile -e '' produces a file
MasterDuke same with --profile-stage
timotimo hm
it didn't get as far at all before, right? 12:32
MasterDuke it did. before when i only had 8gb of ram the oom killer would kick in. but it's been doing this (exit without writing anything) for a long time 12:34
timotimo hmm 12:36
MasterDuke ugh, merge conflicts trying to pull your latest changes. probably thrown off my my added ops. will have to de-conflict later 12:41
but re profiles not being written, i think MVM_profile_end isn't being called. i added some fprintfs a while ago, but never got back to where things broke down 12:42
timotimo ending the profile will also use a lot of memory 12:44
so if it just exits instead of doing the write-out, the total memory usage will be a bunch lower 12:45
Geth_ ¦ problem-solving: lizmat assigned to jnthn Issue "Sequence" means too many things in Raku github.com/Raku/problem-solving/issues/175 12:48
12:52 squashable6 left 12:53 squashable6 joined 12:55 squashable6 left 12:57 squashable6 joined 13:00 gfldex joined
gfldex multi sub postcircumfix:<{ }>( \SELF, Whatever ) does not delegate to AT-POS. That makes it impossible for custom subscripts to deal with $foo{*}. Is that by design? 13:01
lizmat you mean AT-KEY ? 13:02
gfldex: but it does, indirectly? 13:04
multi sub postcircumfix:<{ }>( \SELF, Whatever ) is raw {
SELF{SELF.keys.list};
}
gfldex that does not end with AT-KEY 13:05
jnthn It ends up with an AT-KEY call for each key 13:06
gfldex lizmat: example: //gist.github.com/e8fbee8cbb0c0df2d7b20747e1bd4dcc
jnthn m: class Foo { method AT-KEY($x) { say "AT-KEY $x" }; method keys { <a b c> } }; Foo.new{*} 13:07
camelia AT-KEY a
AT-KEY b
AT-KEY c
gfldex jnthn: I'm shooting for a custom subscript.
jnthn gfldex: If you want to override slicing semantics, you need to export your own `postcircumfix:<{ }>` candidates for your type
gfldex jnthn: that would be an ENODOC in docs.perl6.org/language/subscripts#Custom_types 13:08
jnthn Then please make a doc issue.
gfldex will do
jnthn Thanks
And yes, the design is so that you can just implement AT-KEY and not care about all the slicing stuff, since that's the common case. 13:09
lizmat jnthn: you could argue that AT-KEY(*} should be handled as well, as it is a single entity 13:11
but practically that would be ... eh... not so nice 13:13
jnthn You can argue, but you're not going to win :P
MasterDuke jnthn: any further thoughts on github.com/Raku/nqp/pull/612 ? and/or the related github.com/MoarVM/MoarVM/pull/1267 ? 13:15
[Coke] tried a "make -j" in nqp and the jvm build barfed. guessing it's no longer -j safe. 13:23
(a rebuild with just 'make' was fine)
MasterDuke i'm not sure it's ever been -j safe (in my experience at least) 13:24
Geth_ nqp/jvm-sayfh-rm: 0fbca34681 | Coke++ | 2 files
Remove sayfh opcode

Resolves #614		 13:25
nqp: coke++ created pull request #615:
Remove sayfh opcode
[Coke] MasterDuke: I spent a lot of time at one point trying to make sure everything was -j safe, but honestly it was pre-christmas. 13:26
Geth_ rakudo: Kaiepi++ created pull request #3625:
Add an iterator multi candidate to Supply, make Supply.list 20% faster
[Coke] and it predates at least one overhaul of the build systems. :)
MasterDuke would be nice it was though, it's such a long build 13:27
[Coke] wonders if Geth_ needs to be Geth
so there's no automatic testing for that nqp branch in rakudo, correct? 13:28
MasterDuke correct
[Coke] fires up a manual test. 13:32
timotimo MasterDuke: i'd be interested to hear what perf has to say about the garbage collection times for marking the call graph 13:47
one random measurement i made showed it just below 50% 13:48
MasterDuke timotimo: btw, here's some output with your new commits gist.github.com/MasterDuke17/625d5...e92c3ac3d7 13:49
running under perf now
timotimo doesn't need to run for the entirety 13:50
13:51 tbrowder left, tbrowder joined
timotimo it'd have been cool to have output without the change to the growth strategy 14:03
14:04 sena_kun joined 14:06 Altai-man_ left
[Coke] bartolin_: should we have a ticket for removing the deprecated Unsafe API? 14:07
Kaiepi wait... 14:11
make -j works with nqp and rakudo now? i thought that borked things
maybe that's just with bsd make, which doesn't get used anymore when i build on my system 14:12
with bsd make there are special targets and prerequisites you may need to set in order for make to work in parallel, iirc those don't get used 14:15
[Coke] Kaiepi: no, it *doesn't* work. It *used* to, ages ago 14:32
Kaiepi yeah, i get that, i thought it never worked 14:33
i'll uninstall gmake and check if it was bsd make that was the problem 14:34
would be nice if you could tell the configure script which type of make you want to use though... 14:37
[Coke] make install on rakudo-j soooo slow
I think you can do that with --moar-option 14:38
Moar takes a --make option. 14:39
so from rakudo if you're building everything, you can do perl Configure.pl --moar-option="-make=gmake", I think.
Kaiepi ah, i thought that would only make moarvm use the make passed 14:40
bsd make works ok with -j, this must've been before the big build system changes from a while back
[Coke] I think when nqp runs, it checks moar's config for build options, and then rakudo checks nqp. I think. 14:41
The build system has been entirely rewritten at least once since I touched it. :) 14:42
Kaiepi doesn't seem to do that with ./Configure.pl --moar-option="--make=make" 14:43
on rakudo
[Coke] Apologies. 14:44
"but if moarvm can do it, surely nqp & rakudo can."
(maybe just try --make=make on rakudo's Configure.pl ?) 14:45
14:54 Xliff left, committable6 left 14:55 committable6 joined
vrurg Actually, there is no way to force a particular make for Rakudo and nqp. Neither it should be taken from moar because it's not the only backend. But otherwise this must be fixed. 14:56
[Coke] Should be possible to migrate the existing --make option up to the other Configure's
14:57 softmoth joined
vrurg [Coke]: yes and no. Rakudo -> NQP – sure, yes. But might be questionable if makes sense propagating it down to moar. 14:57
[Coke] It's not being propagated now, but you can specify options to pass to moar when you config rakudo. I suspect it would extend that, you could specify the make for each build, and pass them down if you asked for it from rakudo. 14:58
(or maybe just another indication we should split up the config for rakudo to not build nqp, and nqp not to build moar.) 14:59
14:59 squashable6 left
[Coke] 'make j-test' is hanging. 14:59
15:02 squashable6 joined
vrurg [Coke]: I'll consider it. 15:03
15:07 softmoth left 15:29 softmoth joined
nine It just occured to me: with a generic mechanism to replace $*DISPATCHER and $*NEXT-DISPATCHER, we could bring back the calling context. Wouldn't that be great! 15:36
15:59 softmoth left 16:03 Altai-man_ joined 16:06 sena_kun left
timotimo nine: do you have the opportunity to try profiling your program again? i have a branch (that i should merge mastrduke's branch into as well) that will reduce memory consumption during profiling, and also offer a debughelper that'll tell you (us) what takes how much space 16:11
nine timotimo: not right now :/ Turned out that the huge Inline::Perl5 refactor I did last week didn't do so well in production. My find_method found Any's methods like `first` even when a Perl 5 base class provided such a method. The fix turned out to be quite elaborate including proper support for multiple inheritance 16:24
I've been fighting one obscure error after another today :/ 16:25
A really great one was: 16:26
m: class Bar { }; class Foo is Bar{ }
camelia 5===SORRY!5=== Error while compiling <tmp>
Syntax error while specifying a parent class:
Must specify a space between Bar and {
at <tmp>:1
nine What?!
timotimo oh no :(
nine My Rakudo gives me: class GLOBAL Perl6::Metamodel::PackageHOW cannot have attributes, but you tried to declare '$.name'
timotimo m: class Bar { }; class Foo is Bar { } 16:27
camelia ( no output )
timotimo well, that's odd
nine m: class Bar { }; class Foo is Bar{ has $.name } 16:28
camelia 5===SORRY!5=== Error while compiling <tmp>
A class cannot have attributes, but you tried to declare '$.name'
at <tmp>:1
------> 3s Bar { }; class Foo is Bar{ has $.name 7⏏5}
expecting any of:
constraint
nine Ah, there it is!
A class cannot have attributes.... yeah, right
timotimo haha what :D 16:29
beautiful
16:51 MasterDuke left 17:23 MasterDuke joined
moritz need a space before the { 17:52
timotimo yeah 17:55
MasterDuke timotimo: i did `perf record -g --call-graph dwarf` of the compile + --profile-compile and then `perf report --call-graph=none --no-children` of the resulting 52gb perf data....`95.42% moar libmoar.so [.] MVM_profile_instrumented_mark_data` 18:02
but afk for a bit 18:03
timotimo LOL 18:04
18:04 sena_kun joined 18:06 Altai-man_ left 18:18 MasterDuke left 18:54 MasterDuke joined
MasterDuke timotimo: any new optimizations up your sleeve? something about changing pointers to indices? 19:07
19:16 MasterDuke left 19:30 MasterDuke joined 19:37 lichtkind joined 19:50 sjn_ is now known as sjn 19:52 pmurias left 20:03 Altai-man_ joined 20:06 sena_kun left 20:41 Kaiepi left
MasterDuke timotimo: i added fprintfs in dump_data, MVM_profile_instrumented_end, and MVM_profile_end, but none printed. i also turned on telemeh logging, last two lines are: `5599b5e950b0 5946769630404 -|- "gc finished"` and `0 5947155330840 -) "moarvm teardown" (1)` 20:59
and a profile--compile of CORE.e does successfully get written 21:02
21:24 Kaiepi joined 21:47 tobs left 22:04 sena_kun joined 22:06 Altai-man_ left
AlexDaniel sena_kun: what about now? I'd really like to see the results to add more stuff :) 22:13
22:27 lichtkind left 22:33 lichtkind joined 22:45 tobs joined
sena_kun AlexDaniel, can't build an image with `===> Testing [FAIL]: Pastebin::Gist:ver<1.004002>`. :/ 23:28
AlexDaniel sena_kun: it's an old issue. Can you ignore it? 23:29
23:33 sena_kun left
AlexDaniel hm, actually, the module should be working 23:50
I'm staring at the thing and can't quite figure out what's wrong. The test file creates a gist, deletes it and then checks that it was deleted
and it was, except that somehow it fetches it correctly after deleting it… huh? 23:51

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!