dogbert11 the odd thing is that I don't think the SEGV shows up if the NFG check is off 00:02
samcv ok i think i may have fixed it. well. i think you might be right they are separate 00:03
well. or maybe not completely. anyway there was issues 00:04
dogbert11 two different problems?
samcv well maybe differet problems. that just got exposed by the checks 00:05
dogbert11 aha, and you've fixed one? 00:06
samcv well i fixed that NULL problem in re_nfg that was one. and the NULL in the NFG_debug and uh 00:07
so i am fixing things hah
dogbert11 and the KMP fix as a bonus :) 00:08
samcv and that
you should try that sometime the clang option i said
super cool.
not as slow as valgrind
though it does kill the program when it finds a problem. but it gives a lot of info
and there's a problem that the result of concatenation was getting freed during concatenation 00:09
and i'm gonna push that shortly
dogbert11 cool
Geth MoarVM: 3ca3020705 | (Samantha McVey)++ | src/strings/ops.c
Fix GC problem during concatenation, MVMROOT `result`

Also move the debug checks to inside of the MVMROOT.
In MVM_string_replace(), use MVMROOT instead of manually pushing and popping to root.		 00:11
samcv ok now with the clang instrumentation i'm getting the UTF problem :)
since it aborts and prints out great info if it encounters stuff
but there are memory problems with utf8 00:12
dogbert11, gist.github.com/samcv/8b9288c9b718...beb79d7e9c 00:13
at the bottom 00:14
dogbert11 the spectest errors seems to have disappeared with your latest fix :) 00:23
Geth MoarVM: a22f98dbac | (Samantha McVey)++ | src/strings/utf8.c
MVM_string_utf8_decodestream: free the buffer on malformed UTF-8
 00:25
samcv dogbert11, found three missing MVM_free's in MVM_string_utf8_decodestream
and i must go now, be back in like 5 or 6 hours 00:26
dogbert11 nice, will be asleep when you get back
samcv dogbert11, did you try compiling with clang and -fsanitize=address in cflags and ldflags? 00:27
if not try it :)
they also have a data races detector and many others 00:28
see here for info clang.llvm.org/docs/UsersManual.ht...generation
dogbert11 will check it out, don't even think I have clang installed 00:32
Error, MoarVM cannot get Unicode codepoint property for synthetic codepoint -11 00:33
in block <unit> at t/spec/S15-nfg/concatenation.t line 38
samcv i see that too 00:34
wow 00:35
ok MVMROOT of the result caused it
but why
that's... odd
hm
well feel free to revert that
if moarvm needs to get bumped or something. g2g. will check it out when i get back though 00:36
dogbert11 drops to bed since it's 0230 AM
Geth MoarVM: 4eaba7a04b | (Samantha McVey)++ | src/strings/ops.c
Revert "Fix GC problem during concatenation, MVMROOT `result`"

This reverts commit 3ca30207055d7f8ca28071ef20ffcfb88cd82241.
Caused some failing tests in roast.		 01:05
MoarVM: cd41322ec3 | (Samantha McVey)++ | src/strings/ops.c
Cautiously initialize more memory pointers as NULL
 01:15
01:22 greppable6 joined, releasable6 joined, quotable6 joined, bisectable6 joined, committable6 joined, coverable6 joined, bloatable6 joined, evalable6 joined, benchable6 joined, unicodable6 joined, statisfiable6 joined, nativecallable6 joined 01:52 ilbot3 joined 02:24 ilbot3 joined 08:32 lizmat_ joined
samcv .tell dogbert17 check out how the address sanitizer for clang works: github.com/google/sanitizers/wiki/...rAlgorithm very interesting how it works. even though this is how it works, it runs way faster than valgrind probably since doesn't have to run inside and have that overhead 08:42
yoleaux samcv: I'll pass your message to dogbert17.
timotimo samcv: valgrind has the added benefit that you don't have to recompile. which is especially good if you don't have access to the source code of parts or the whole program 08:46
samcv yeah. though the clang one detected problems valgrind never did
and detected problems accessing stuff on the stack as well
but yeah. you don't always have the ability to want to quickly recompile 08:47
this is their comparison github.com/google/sanitizers/wiki/...emoryTools 08:48
optionally you can have t do stack-use-after-return. that's neat 08:49
in case you return an object on the others stack and then you shouldn't access it
09:14 dogbert11 joined
dogbert11 samcv, the revert made the number of failing spectests go up significantly 09:22
timotimo time to move this work off to a branch so we can flatten eventual reverts and such away when we merge? 09:31
oh
it only crashes on travis master m(
things we should have: dup and maybe dup2 for IO::Handle. also, creating IO::Handle.from-fileno or .from-native-descriptor 10:58
12:07 dogbert11 joined
nine I can now JIT native calls with one uncontainerized int64 or CPointer argument :) 12:16
timotimo nice 12:20
that's already pretty useful 12:21
when you get two, that's a big step into useful territory
nine I think deconting is more important even 12:22
timotimo true
nine At least if I ever want to merge this thing ;)
I'm just not sure in which layer exactly to do that. 12:23
timotimo if we emit the decont in spesh we can turn it into simpler operations in some cases 12:24
like known type and such
nine Yeah, I guess it's best (and probably easiest, too) to decont in the generated routine body when binding params to locals. Right now I access the args buffer of the callsite directly. 12:26
So a bit of work to change that, but probably not really difficult.
Geth MoarVM/jit_nativecall: ac4d23f9f3 | (Stefan Seifert)++ | 3 files
JIT native calls with an unconted int64 or CPointer argument

Doesn't handle containers yet as it neither checks nor deconts.		 12:42
12:46 AlexDaniel joined 12:47 committable6 joined, benchable6 joined, evalable6 joined, quotable6 joined, unicodable6 joined, bloatable6 joined, bisectable6 joined, coverable6 joined, greppable6 joined, releasable6 joined, statisfiable6 joined 13:03 lizmat joined 13:52 greppable6 joined
nine So what's the correct way to get a deconted param into a local? 14:08
lizmat has no idea 14:09
timotimo regular param op and a decont after that i'd say 14:10
nine Like this? QAST::Var.new( :name($lowered_name), :scope<local>, :decl<param>), QAST::Op.new( :op<bind>, QAST::Var.new(:scope<local>, :name($lowered_name)), QAST::Op.new(:op<decont>, QAST::Var.new(:scope<local>, :name($lowered_name))),) 14:12
timotimo i think so 14:18
nine Yes, looks good :) Now I only hav eto change the JITed code to access locals instead of params 14:19
timotimo cool
nine Would be easier though if I weren't as tired 14:20
samcv dogbert11, are you sure they went up? 14:22
i just woke up i can run again but i remember running them again 14:23
dogbert11 hello samcv :) 14:43
ok, I'll check out master and try again ... 14:44
so I'm at the S04 tests now and there's been complaints from 20 files already 14:48
ofc they could all be errors uncovered by the NFG checker
samcv i have that off right now 14:49
so it must be the NFG checker
i think it's probably good if we set all pointers to start as NULL. and the compiler will always optimize it out or otherwise the performance difference is basically none
dogbert11 sounds like a plan, btw have you had your tea yet? 14:50
samcv though it should get compiled out if there's no way for it to be used, if it's okay i will start commiting more pointers to be initialized as NULL
no!
dogbert11 uh oh
samcv yep
dogbert11 FWIW I rambled a bit about the Malformed UTF8 problem over in the dev channel 14:52
samcv i get full spectest pass
dogbert11 without the checker so do I
timotimo i know how to make this faster! i'll build a binary format! 14:53
-- 10 hours later --
samcv dogbert11, but you say that with my reversion more tests fail (with the NFG checker) than before? 14:56
nine /win^H^H^H^H^H^H^H^H^H^H/win 10 14:58
samcv for some reason if i ./Configure.pl --asan then i can't run it. does it include its own version of asan?
cause if i put -fsanitize=address manually it works fine 14:59
well i'm going to try running spectest with asan and nfg check on and see if it catches anything 15:02
dogbert11 let me try without the revert 15:03
dogbert11 rebuilds
if I run asan with gcc I have to use that option 15:05
samcv yeah. probably since we must include the library. but clang has its own never version and otherwise i get a mismatch
ugh why do i get failing tests with `make spectest` but not running them by themselves 15:06
dogbert11 am at S06 now, only one fail so far ...
nine timotimo: looks to me like spesh will quite happily throw my deconts away as it doesn't know that the nativecallinvokejit op will access the locals. Any idea how to avoid that?
timotimo oh 15:07
yeah that's a problem
dogbert11 samcv: the synthetics errors we saw yesterday, could those have been legit?
samcv uh corruption 15:08
idk if that's legit or nonlegit?
but probably something being freed when it shouldn't
dogbert11 only five fails now 15:09
timotimo nine: short of inventing an op that does nothing except touch a variable ... :\
i mean, we could put the args into the nativecallinvoke op's signature, but we'd have to have a whole bunch of these ops
samcv oh well i guess maybe it could be legit idk hard to tell. but really. it seems odd...
i'm getting lots of failures but some of them i would run the test manually and pass fine 15:10
dogbert11 samcv, now t/spec/S15-nfg/concatenation.t fails with error, 'MoarVM cannot get Unicode codepoint property for synthetic codepoint -11'
samcv yep
dogbert11 do you think that's due to corruption or could it in fact be legit? 15:11
samcv oh it could be legit idk... it was super confusing to me yesterday 15:12
dogbert11 let's take another look, sec
samcv yeah i made several changes in that one commit. i'm going to make one at a time
nine timotimo: or teach spesh about nativecallinvokejit touching locals? 15:13
timotimo okay, but which does it touch? 15:14
you can teach facts.c about that op
but you need to find the right register and version for this to work
nine timotimo: well version 2 of all locals in that frame 15:15
timotimo if we're willing to rely on this ...
dogbert11 samcv: here's gdb output gist.github.com/dogbert17/b6a5ed34...ef9868f116 15:16
timotimo so ... you know how i made that binary format to make moar-heapanalyzer faster? 15:17
it's slower by a factor of like 6
nine For the first implementation of this I'd say yes.
timotimo ok, so in facts.c you can, when you stumble upon a nativecallinvokejit, bump up the usages of all locals of version 2, i.e. for loop from 0 to num-locals from somewhere in the spesh graph 15:19
dogbert11 this is the failing test: is-deeply ('a' x 2 ~ "\c[COMBINING ACUTE ACCENT]" x 3).ords, (97, 225, 769, 769), "Concat works with combining repeated characters";
nine timotimo: yep, that's pretty clear. But not where the facts for the locals are actually 15:20
There's num_locals in MVMSpeshGraph, but I can't find any list representing locals 15:21
timotimo you go through the fact functions
oh, you pass them a MVMSpeshOperand
no, actually, i think you pass register and version separately
so no problems at all there 15:22
samcv dogbert11, that fails without it reverted?
yeah i'm getting that too 15:23
dogbert11 let me check that
with the revert in place it does not fail 15:24
timotimo 4:46 to get a summary of the snapshot in my new version; 42s in the old version (with compatibility changes that might make it slower) 15:26
nine timotimo: like case MVM_OP_nativecallinvokejit: for (i = 0; i < g->num_locals; i++) { g->facts[i][2]->usages++ } break;? 15:27
timotimo yeah, try that 15:30
nine or rather [i][1] as I guess we start counting at 0? :) 15:31
timotimo could very well be, or maybe we only output it like that for human readability? 15:32
nine No change so far... 15:33
LOL setting MVM_SPESH_LOG "fixes" the issue 15:34
timotimo somehow the C recompile got messed up perhaps?
15:35 AlexDaniel joined
nine No...for some reason it just starts to spesh later and thus never. Upping the ei^H^Hiteration count helps 15:35
timotimo :buffer on IO::Handle won't work for :bin? 15:38
i.e. it's still doing the exact reads as syscalls that i'm calling from my hll code
nine According to the spesh log, the usages are still all 0 15:41
samcv ok i found the problem dogbert11 15:47
dogbert11 wee
what was it?
samcv gonna run spectest. but 15:48
it was running NFG_CHECK_CONCAT when it shouldn't have. i.e. when it was going to return re_nfg anyway because the concatted string wouldn't have been correct 15:49
though still dunno why it would say something about a negative grapheme but. that was the cause
gonna run spectest to make sure
dogbert11 I'll wait :)
samcv ugh i wish asan wouldn't cause spectest's to fail for no reason then i run them again and it's fine. they show that they didn't run any tests, so i'm not sure what happened. how can i run tests but show full stderr? 15:51
dogbert11 I would like to know that as well 15:53
samcv ugh getting errors 15:54
well from asan
on make m-spectest
well starting it
yeah this is the same issue i had before and why i added `result` to the MVMROOT. let me add that again 15:55
ok cool. fixed it now by adding that :) 15:56
nine timotimo: ha! ase MVM_OP_nativecallinvokejit: for (i = 0; i < g->num_locals; i++) g->facts[i][g->fact_counts[i] - 1].usages++; break; 15:58
samcv dogbert11, this is my fix though github.com/MoarVM/MoarVM/commit/08...97db7a9de5 15:59
timotimo CurrentThreadScheduler is a godsend 16:00
why are you going to the facts array directly?
oh you were doing that before
nine timotimo: I dare say this is actually even pretty safe. Worst case it prevents some instruction elimination.
timotimo why didn't i yell at you
nine timotimo: tweak_block_handler_usage does it, too 16:01
timotimo hm.
naughty naughty
nine So containers are now handled which gives us full access to CPointer arguments. Unboxing however isn't yet and the deconting code seems to have broken even the workaround (passing a native int) 16:03
samcv ugh --verbosity doesn't seem to work? set it to 10 and get only progress bars. ]set to 1 and i get only progress bar 16:08
timotimo 2:46m now instead of the 442s
samcv set to 0 and it is the same as it is normally
dogbert11 perhaps it's not fully implemented 16:09
samcv Zoffix, any help? 16:10
zoffix always knows secrets with roast
idk if it's skipping these tests because they are taking a long time to run? i get "no subtests run", if i run manually it takes a while for it to start outputting stuff at all. so maybe it's just killing it when it doesn't get output for a period? 16:11
(harness5 here)
dogbert11 are you running with asan? 16:13
also, harness6 is broken so we have no alternatives atm :)
samcv yeah with asan
timotimo someone want to implement buffered binary reading? :) 16:14
dogbert11 I sometimes forget that mem requirements double when usin that switch
*using
timotimo maybe i should ask in #perl6, as this is a hll task
samcv i think it's slower too cause i have it set to `detect_stack_use_after_return=1 ` as well
i think that slows it down
dogbert11 no swapping?
samcv how do i fix "Unhandled exception: Missing or wrong version of dependency " 16:35
i mean when running my system's perl6 inside the rakudo git folder
i want it not to fail like that 16:36
is there some env var?
also i hate this error :P
16:49 lizmat joined
dogbert11 samcv: have you tried rebuilding rakudo? 16:55
sry, didn't read properly 16:56
Zoffix samcv: help with what?
samcv oh. getting make spectest to be verbose
i want to get all stderr messages to my terminal 16:57
i haven't been able to do it by setting --verbosity but i may be doing it wrong or setting it to the wrong number
Zoffix samcv: no idea. If you use t/fudgeandrun file1 file2 it'll be verbose. But I don't know an easy way to run all tests verbosely 16:58
nine Ok, integer arguments are just not there yet due to missing unboxing support, but I do support an arbitrary number of pointer arguments now :)
Zoffix samcv: did you try --verbosity=1 ? 16:59
samcv i think i did. will try again though
Zoffix samcv: looking at available values here: metacpan.org/pod/TAP::Harness#Class-Methods
timotimo yay 17:00
new heapanalyzer down to taking "only" 1:16m vs old 38s, but the memory usage is down to 450mb from 1.8gb
Zoffix \o/ 17:01
GitHub-- rendering POD instead of showing the script: github.com/rakudo/rakudo/blob/nom/t/harness5 17:05
samcv: ^ if you edit that file and change `merge => 1` to `merge => 0` and run the following, then you get STDERR on terminal: /home/zoffix/perl5/perlbrew/perls/perl-5.26.0/bin/perl t/harness5 --verbosity=1 --fudge --moar --keep-exit-code --tests-from-file=t/spectest.data
AlexDaniel xD
samcv oh nice! 17:06
Zoffix No idea how to make it just normal verbose output with all the things
Zoffix goes back lurking
timotimo whoops, double-freeing all the stuff
turns out just nativecasting to Pointer will not mark the Pointer for "will not be freed" 17:07
how do i even nativecast 17:08
samcv hmm it seems when re_nfg is ran, if the string is utf8-c8, it comes out with a different number of graphemes afterward 17:25
timotimo does it misunderstand the special graphemes we make for our c8 thingies? 17:26
samcv idk i assumed it would
timotimo me too 17:27
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); $u.=subst("a","b"); say $u
camelia g????xF8o????xD9????xD2b?
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); $u.=subst("a","b"); say $u.chars
camelia 9?
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.chars; $u.=subst("a","b"); say $u.chars
camelia 6?9?
samcv uh
m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.chars; $u.=subst("a","a"); say $u.chars 17:28
camelia 6?9?
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.codes; $u.=subst("a","a"); say $u.codes
camelia 15?15?
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.ords; $u.=subst("a","a"); say $u.ords
camelia (103 1114109 120 70 56 111 1114109 120 68 57 1114109 120 68 50 97)?(103 1114109 120 70 56 111 1114109 120 68 57 1114109 120 68 50 97)?
timotimo what's with these ords
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.comb; $u.=subst("a","a"); say $u.comb
camelia (g ????xF8 o ????xD9 ????xD2 a)?(g ????xF8 o ????xD9 ???? x D 2 a)?
samcv well it's utf8-c8
timotimo but still
samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.comb.join(', '); $u.=subst("a","a"); say $u.comb.join(', ') 17:29
camelia g, ????xF8, o, ????xD9, ????xD2, a?g, ????xF8, o, ????xD9, ????, x, D, 2, a?
samcv so yeah it's losing the utf8-c8 graphemes
poop 17:30
timotimo, maybe if it loses the graphemes, it can still write as utf-c8 properly?
17:30 benchable6 joined
samcv but still you colud then replace the letters that make it up like replace all D's with something and ruin the utf8-c8 17:30
ah it iterates by codepoint 17:31
poop poop 17:32
well i opened a moarvm issue 17:40
timotimo heapanalyzer rewrite at 1:10 vs 42s 17:52
1 minute! 18:04
35 seconds! 18:11
25 seconds! 18:14
(but i have a bunch of nqp ops in there)
18:23 Ven`` joined
Geth MoarVM/heapsnapshot_binary_format: f28600e499 | (Timo Paulssen)++ | src/profiler/heapsnapshot.c
introduce a binary format for heapsnapshot

it starts with a version identifier to differentiate between the old format and this one. all records except the string heap are fixed sizes and the file ends in a table outlining the sizes of the individual segments.
each segment starts in a little identifier to make sure the snapshot parser doesn't read bogus data, and before a set of entries there's an entry of the entry size.		 18:28
timotimo this was my idea of fun for the day ...
i should have probably written into the file that snapshots end up a noticable bit bigger 18:29
i'll amend
Geth MoarVM/heapsnapshot_binary_format: 7960ecc26e | (Timo Paulssen)++ | src/profiler/heapsnapshot.c
introduce a binary format for heapsnapshot

it starts with a version identifier to differentiate between the old format and this one. all records except the string heap are fixed sizes and the file ends in a table outlining the sizes of the individual segments.
... (7 more lines)
timotimo not sure fi i can get it faster than the 20 seconds i have now 19:07
maybe if i split the collectables in half as well, not only the references 19:08
this code will want a bit of cleanup for clarity when i'm done making it fast
17 seconds! 19:21
pushed all changes to the heapanalyzer 19:33
19:38 AlexDaniel joined
timotimo it looks like the amount of ram claim is wrong, or at least i threw out the benefit by parallelizing harder 19:42
samcv so this lets us dump the memory heaps or what 19:54
timotimo it's a step in the right direction 20:00
samcv or it analyzes them? give me a rundown?
nice
timotimo it's two parts 20:01
one is the new format that moar can spit out
it can get by completely without MVM objects and hll code
the other part is support for this format in the heap analyzer
it's around 2x faster, but uses more memory
that ought to let us work with stuff a bit faster
one other thing that would be nice would be to maybe parallelize the breadth-first search 20:02
does that make sense? 20:03
the heapsnapshot_binary_format branch will just unconditionally write to /tmp/heapsnapshot_new_format whenever a heapsnapshot is finished 20:04
AlexDaniel: was it you who was interested in faster heap snapshot loading? 20:05
i know it was a whateverable developer :D 20:06
samcv nice 20:11
timotimo next on the feature request list is being able to get all-paths to nodes
samcv looks like we have a heap-buffer-overflow 20:12
addresssanitizer is angry 20:13
244 bytes to the left of the closest area
looks like is_stack_frame() is checking a stack frame's struct value, but that's actually not inside any mapped memory 20:14
timotimo, MVM_gc_allocate_nursery() is that run on start and not like, afterward?
cause that's a calling function in the chain
timotimo now i'm wondering if i should always generate an allpaths or only sometimes 20:17
the "all paths" for type objects can get hilariously big 20:18
i could either run a bfs and note down all predecessors and just stop adding more after reaching 10 or something, or i do a new DFS each time an all-paths is asked for 20:22
i'll try DFS in a gather/take and let the user stop it whenever they want to 20:25
samcv wow so far the address saniiser hasn't puked when running roast. yay 21:19
timotimo cool 21:20
i'm not being very good at the whole "build a dfs" thing
samcv so this is in perl6 then, not C? 21:22
timotimo it is
samcv it hung somewhere but it ran 228 tests and all pass 21:23
timotimo okay the dfs for a single object is currently 83k steps away from the root and is still going 21:28
it's nice that i print " " x $length-of-that-path before each debug output 21:29
samcv heh 21:30
timotimo oops 21:31
i was looking at the size of the queue
it looks like there's a load of gc pauses every now and then but it shouldn't be creating many objects that live long 21:33
i'll go to bed i think 21:37
lizmat too 21:43
good night,! 21:44
Zoffix 3
Night
21:54 zakharyas joined
jnthn . 21:58
yoleaux 26 Aug 2017 16:31Z <japhb> jnthn: website typos: First HTTP flip page: 'SITINT', last HTTP flip page: 'requset', first Tooling flip page: 'delcaring'.
14:35Z <AlexDaniel> jnthn: FWIW, remember I asked how to process null-separate output? Turns out there's an identical problem when trying to process \n-separated output. See github.com/perl6/doc/issues/1472 and rt.perl.org/Ticket/Display.html?id=131923
22:02 committable6 joined 22:12 dogbert11 joined
Geth MoarVM: samcv++ created pull request #665:
Fix GC issues and misc. * Make sure MVMROOT'd pointer is initialized as NULL * Can cause issues if GC runs while the pointer is uninitialized.		 23:22
samcv ok yay. if travis says it's good then all issues should be good 23:23
memory issues are the least fun part of programming in C :P 23:24
but now AddrSanitiser is clean of any errors of improper accesses on the stack or allocated space etc 23:25
err roast that is
Geth MoarVM/master: 6 commits pushed by (Samantha McVey)++ 23:42
samcv jnthn, i've been thinking of how we can fix codepoint iterators when renormalizing and retain the UTF8-c8 graphemes. not sure if we should change the codepoint iterator to either have an option you can set to allow it to hand back synthetics (but only utf8c8 synthetics) so they pass them along. and most of our things will assume they're control because they're less than 20 23:44

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!