00:01 JRaspass left 00:03 sena_kun joined 00:05 Altai-man_ left
timotimo so i've been playing around with lembark's workload and got heaptrack to actually output something helpful 00:14
120629739 calls to allocation functions with 88.41GB peak consumption from
[...]
8783964 calls with 83.13GB peak consumption from:
uv__read
that looks like the program is either holding on to all the byte arrays that are being created by reading from the child process 00:15
or perhaps the child process is just writing as fast as it can go, which makes a bigger and bigger queue of read data pieces that we're too slow to process 00:16
00:19 Voldenet left 00:20 Voldenet joined, Voldenet left, Voldenet joined
timotimo well, i've sent sigstop to the zxcat process that's feeding data into the rakudo process and it's still going 00:30
i haven't looked at the source of the workload program much yet, so i'm not sure what exactly the status output lines mean 00:31
the memory size is sloooowly growing, i'm not sure how the growth would have been during the same timeframe but without stopping the zxcat process
so maybe that's where i should look next
also i should make sure the request-garbage-collection isn't causing objects to be promoted to the old generation too fast and thus wasting gen2 space that won't "shrink back" 00:34
the input data is about 123 GiB uncompressed 00:38
so if rakudo is processing slow enough and xzcat is spitting out fast enough, i can imagine rakudo going over 128 GiB RSS quite quickly 00:39
oh bother i didn't "nohup" the task, and there's neither tmux nor screen installed either 00:53
it's consumed about 1.3k minutes of cpu time and it's just recently started writing out the "stubs" of the chunks, meaning it's finished "process-chunk" on a bunch of the threads and is now sorting hashes to write to the files 00:58
about 32k entries each per chunk it seems like
at this point top shows the RES to be 4.8 gigs 01:00
m: say 37.base(16) 01:01
camelia 25
timotimo four more threads until all threads have passed on to stage "sort and output to file" 01:02
m: my @pieces = ["a".."z"].roll(128) xx 33000; say @pieces.elems; my $start = now; @pieces .= sort; say now - $start 01:03
camelia 33000
3.7065022
timotimo m: my @pieces = ["a".."z"].roll(128) xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ $^a.substr(0, 64) <=> $^b.substr(0, 64) or $a.substr(64, 64) cmp $b.substr(64, 64) }); say now - $start 01:04
camelia 33000
Cannot convert string to number: base-10 number must begin with valid digits or '.' in '3⏏5d s d m f l r w w j v j w i l c w i b i t p n w j v m l y g f m ' (indicated by ⏏)
in block <unit> at <tmp> line 1
timotimo m: my @pieces = ["a".."z"].roll(128).join("") xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ $^a.substr(0, 64) <=> $^b.substr(0, 64) or :16[$a].substr(64, 64) cmp :16[$b].substr(64, 64) }); say now - $start
camelia 33000
Cannot convert string to number: base-10 number must begin with valid digits or '.' in '3⏏5emmqkzndudltwxhkrilqjqjeffmroglztuwoozhjojyicscogaqdneakxgiemppl' (indicated by ⏏)
in block <unit> at <tmp> line 1
timotimo oh, it's not even base 16 silly me 01:05
m: my @pieces = [|"a".."f", |"0".."9"].roll(128).join("") xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ :16[$^a].substr(0, 64) <=> :16[$^b].substr(0, 64) or $a.substr(64, 64) cmp $b.substr(64, 64) }); say now - $start
camelia Potential difficulties:
To apply a Slip flattener to a range, parenthesize the whole range.
(Or parenthesize the whole endpoint expression, if you meant that.)
at <tmp>:1
------> 3my @pieces = [7⏏5|"a".."f", |"0".."9"].rol…
timotimo m: my @pieces = [|("a".."f"), |("0".."9")].roll(128).join("") xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ :16[$^a].substr(0, 64) <=> :16[$^b].substr(0, 64) or $a.substr(64, 64) cmp $b.substr(64, 64) }); say now - $start 01:06
camelia 33000
Cannot convert string to number: trailing characters after number in '03249⏏5b3469b21abd139d913e9b2607eeced66f19c56cc6c1fb6ed18858d5957ff62e138268847dbf8a7288a7ed55f186335db3e32bec36c3015d7ad5e823a31df7' (indicated by ⏏)
in block <u…
timotimo i'm using :16[] wrong, aren't i
m: my @pieces = [|("a".."f"), |("0".."9")].roll(128).join("") xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ parse-base($^a, 16).substr(0, 64) <=> parse-base($^b, 16).substr(0, 64) or $a.substr(64, 64) cmp $b.substr(64, 64) }); say now - $start
camelia (timeout)33000
timotimo m: my @pieces = [|("a".."f"), |("0".."9")].roll(128).join("") xx 33000; say @pieces.elems; my $start = now; @pieces .= sort({ parse-base($^a.substr(0, 64), 16), parse-base($^a.substr(64, 64), 16) }); say now - $start 01:07
camelia 33000
3.8476713
01:08
timotimo two-arg sort is slooooowwwww
i think i should really recommend lembark to change that 01:09
i sigcont'd xzcat and the rakudo process immediately started growing again, stopped it again and it stopped growing 01:13
how exactly is run(...).out implemented? 01:16
is it "just" a react/whenever that appends to some buffer or channel as soon as it gets data?
anyway it's about time for me to go to bed; none of the threads has finished the sorting bit yet 01:18
01:23 JRaspass joined
jdv79 timotimo: interesting 01:36
02:02 Altai-man_ joined 02:04 sena_kun left 02:43 AlexDaniel left 02:44 AlexDaniel joined 02:45 AlexDaniel left, AlexDaniel joined 02:55 MasterDuke left 03:29 Kaiepi joined 04:03 sena_kun joined 04:05 Altai-man_ left 04:40 JRaspass left 05:01 nebuchadnezzar joined 06:02 Altai-man_ joined 06:04 sena_kun left 07:45 squashable6 left, squashable6 joined 07:51 JJMerelo joined 08:03 sena_kun joined 08:05 Altai-man_ left 08:25 leont joined
lizmat Files=1307, Tests=113021, 217 wallclock secs (28.62 usr 8.34 sys + 3026.18 cusr 286.18 csys = 3349.32 CPU) 08:39
[Tux] Rakudo version 2020.06-52-ga74550507 - MoarVM version 2020.06-20-g187b4564e
csv-ip5xs0.853 - 0.854
csv-ip5xs-207.910 - 8.418
csv-parser26.629 - 27.471
csv-test-xs-200.386 - 0.403
test7.622 - 7.961
test-t1.920 - 2.025
test-t --race0.861 - 0.950
test-t-2032.082 - 33.193
test-t-20 --race9.314 - 9.563
09:00
09:14 AlexDaniel left, AlexDaniel joined 09:16 AlexDani` joined 09:19 AlexDaniel left 09:23 JJMerelo left 09:27 Xliff joined 09:43 JJMerelo joined 09:45 AlexDani` is now known as AlexDaniel, AlexDaniel left, AlexDaniel joined 10:02 Altai-man_ joined 10:05 sena_kun left
Geth rakudo/remove-Seq-nonsense: c9a4f07de2 | (Aleks-Daniel Jakimenko-Aleksejev)++ | 2 files
Remove caching of Seqs

This is a temporary partial commit. If you see it on master then things went very wrong.
10:44
rakudo/remove-Seq-nonsense: 9c19fddc47 | (Aleks-Daniel Jakimenko-Aleksejev)++ | src/core.c/Seq.pm6
Fix rakudo install

Another temp commit. The method is skip-one, not skip. It used to build but not install, now both work fine.
11:29
rakudo/remove-Seq-nonsense: 95cadc8916 | (Aleks-Daniel Jakimenko-Aleksejev)++ | src/core.c/Seq.pm6
Fix a thinko in AT-POS

It's the number of elements, so surely it's larger than the index.
AlexDaniel Altai-man_: can you run Blin on remove-Seq-nonsense branch? I'm expecting a bunch of stuff to fail, I just wanna know how much and how exactly it fails 11:54
Altai-man_ AlexDaniel, will run one tonight.
AlexDaniel thanks
12:03 sena_kun joined 12:05 Altai-man_ left 12:07 JJMerelo left
Geth rakudo: f96a641fff | (Elizabeth Mattijsen)++ | src/core.c/Exception.pm6
Wordwrap X::Adverb message
12:26
rakudo: 1f93724f71 | (Elizabeth Mattijsen)++ | src/core.c/Exception.pm6
Wordwrap X::Syntax::KeywordAsFunction message
rakudo: 98c7e512a6 | (Elizabeth Mattijsen)++ | 2 files
Re-imagine X::Syntax::ParentAsHash message

Sadly, even though this error is a syntax error to be thrown at compile time, it sadly has no access to parser state so it cannot actually produce the line info. Try to make up for it as best as we can for now.
rakudo/rakuast: 1cf5223bd4 | (Jonathan Worthington)++ | src/Raku/ast/regex.rakumod
Compile regex sequences, alterations, conjunctions
12:27
lizmat afk for a few hours& 12:28
Geth rakudo/rakuast: bd890a8813 | (Jonathan Worthington)++ | src/Raku/ast/regex.rakumod
Fix copy-pasta; vendethiel++
13:53
rakudo/rakuast: 55e15dbfe4 | (Jonathan Worthington)++ | 3 files
RakuAST hanlding of implicit $!, $/, and $_

At the compilation unit and routine level.
14:02 Altai-man_ joined 14:05 sena_kun left
Geth rakudo/rakuast: 89921f7ea6 | (Jonathan Worthington)++ | 2 files
RakuAST for regex anchors
14:22
14:36 softmoth joined 14:48 softmoth left 16:03 sena_kun joined 16:05 Altai-man_ left
moritz I'm curious, why are things like RakuAST::SinkBoundary classes instead of roles? 16:22
jnthn moritz: 'cus the node to MOP call translator didn't learn roles yet 16:23
moritz ah, pity :D
jnthn I'll get there eventually :)
It's listed as a "todo" in the README, even :) 16:24
Under the "tasks for anyone wanting to help"
moritz who reads thoss, ever? :D
jnthn Nobody crazy enough, apparently ;)
moritz is that README in rakudo? didn't find it there 16:25
jnthn In src/Raku/ast 16:26
moritz ah
do I need any magic to build? 16:30
currently I get Only typename on RHS of constant implemented so far at line 5850, near "nqp::hash(
(after Configure.pl --gen-moar && make)
sena_kun for the record, has read that section, but is probably too dumb anyway
16:34 squashable6 left
jnthn moritz: Ah, maybe I need to bump NQP_REVISION...it needs an NQP with 30b1d064da767bb 16:35
linkable6 (2020-06-10) github.com/Raku/nqp/commit/30b1d064da Allow more interesting constants in NQP
16:36 squashable6 joined
moritz ok, I'll try with a newer nqp (and moar) 16:36
Geth rakudo/rakuast: f000b5780c | (Jonathan Worthington)++ | 2 files
RakuAST for various built-in char classes
16:38
rakudo/rakuast: 5469d3cf14 | (Jonathan Worthington)++ | tools/templates/NQP_REVISION
Bump NQP_REVISION

So that the rakuast branch builds "out of the box".
rakudo/rakuast: 724c683fca | (Jonathan Worthington)++ | 2 files
RakuAST support for basic regex quantifiers

Along with backtrack modifiers.
17:08
jnthn Almost enough to do 10-regex.t. It uses / <alpha> / which means I need to model assertion syntax too. But not right now...hometime. 17:12
17:40 AlexDaniel left, AlexDaniel joined, AlexDaniel left, AlexDaniel joined 18:02 Altai-man_ joined 18:05 sena_kun left 18:34 MasterDuke joined
timotimo hm, we don't have "int" constants, right? 18:44
in nqp, i mean
lizmat QRegex/Cursor.nqp has this comment in it: 18:53
my $pass_mark := 1; # NQP has no constant table yet
so I guess that is correct
timotimo the mast compiler has a bunch of "constants" too
Geth rakudo/rakuast: 89a19d7d15 | (Moritz Lenz)++ | src/Raku/ast/README.md
Fix typo
18:55
19:22 vrurg left 19:42 vrurg joined
jdv79 timotimo: did you get anywhere excidint with lembark's stuff? 19:53
*exciting
20:04 sena_kun joined, Altai-man_ left
timotimo i haven't touched it again today 20:15
Geth rakudo: 248664c6c3 | (Elizabeth Mattijsen)++ | src/core.c/Exception.pm6
Wordwrap X::Placeholder::Block message

And slightly reword it
20:33
rakudo: 3bc4fa90ad | (Elizabeth Mattijsen)++ | 2 files
Re-imagine X::Syntax::Variable::Twigil message

By providing it with more information, and make it more palatable to readers that read English as a second language, by describing the oxymoron rather than just using the word.
timotimo lizmat: since you're currently already working on exception output, i'd like to toss this your way for a quick look: gist.github.com/egmontkob/eb114294...ck-example 20:44
lizmat looks interesting, doesn't work for me on MacOS, but then again, I'm not on Catalina 20:46
so not bleeding edge 20:47
I guess it could be used for the default .WHY ?
m: say Int.WHY
camelia No documentation available for type 'Int'.
Perhaps it can be found at docs.raku.org/type/Int
»
timotimo hm that's already clickable 20:53
lizmat yeah, because of your IRC client, I presume? It's just text in the code, afaik
timotimo nah, this irc client doesn't do that 20:54
in fact, if it's word-wrapped i have to switch to its special "raw mode" so i can click or copypaste it in full
lizmat "No documentation available for type '$name'. 20:55
Perhaps it can be found at docs.raku.org/type/$name"
hehegithub.com/rakudo/rakudo/blob/mast...Mu.pm6#L86 # nothing magic there 20:56
github.com/rakudo/rakudo/blob/mast...Mu.pm6#L86 # rather
Geth rakudo: 16d24a2127 | (Elizabeth Mattijsen)++ | src/core.c/Mu.pm6
Wordwrap Mu.WHY
21:05
21:23 finsternis joined 22:02 Altai-man_ joined 22:05 sena_kun left
AlexDaniel Altai-man_: .tell me when it's ready 22:36
22:36 AlexDaniel left 22:48 tbrowder left 22:50 kawaii left 22:52 SmokeMachine left, zostay left 22:53 chansen_ left 23:27 lizmat left 23:41 Xliff left 23:43 Geth left 23:44 Geth joined 23:46 Geth left 23:47 Geth joined