samcv i almost forgot how many of the libs i made for the unicode database rewrite that ended up being used for the collation arrays stuff now that i'm working on integrating it into moarvm codebase 00:24
neat
01:55 ilbot3 joined 01:58 geekosaur joined 02:43 geekosaur joined 05:08 brrt joined
brrt good * #moarvm 05:14
05:28 tartal joined 06:12 brrt joined 06:26 leont joined 07:35 leont joined 08:05 domidumont joined 08:06 zakharyas joined 08:18 dogbert2 joined
jnthn morning o/ 08:31
09:01 zakharyas joined 09:07 AlexDaniel joined
brrt morning jnthn 09:27
09:31 zakharyas1 joined 09:42 ggoebel joined 09:49 zakharyas joined 09:51 Skarsnik joined 10:11 dogbert17_ joined 10:17 Voldenet joined
stmuk_ I think moar is broken on windows -- "Unhandled exception: Unable to allocate an array of 8 elements" 10:28
jnthn Weird. Sounds like a job for bissect. 10:29
stmuk_ ok I'll try once virtualbox windows updates ok 10:32
jnthn Yeah, I can't guess where that mighta come from... 10:33
It looks like an out of memory error but...8 elements ain't much
And why would it be out of memory, etc.
timotimo the calculation for how many elements we're allowed to allocate could just be bogus in some corner case 10:34
jnthn Ah, ok
Skarsnik Does Moar know how much total memory is allocated? 10:36
for the thing he allocated I mean
timotimo it asks for that value to figure out when a gen2 gc is appropriate
github.com/MoarVM/MoarVM/blob/mast...ray.c#L350 - stmuk_ can you attach a debugger and break on this exception throw? 10:37
github.com/MoarVM/MoarVM/pull/577/files - it'd still be awesome if this could land soon-ish :S 10:48
Skarsnik hm? 10:49
timotimo a pull request from long, long ago
bringing a feature that's very, very nice 10:50
Skarsnik to be able to bind struct s[]; ?
timotimo something like that yeah 10:51
Skarsnik nativecallable6, struct s {char *a;}; extern struct s foo[];
nativecallable6 Skarsnik, class s is repr('CStruct') is export {ā¤ has Str $.a; # char* aā¤}ā¤our $foo is export = cglobal(LIB, "foo", CArray[s]);
timotimo it'll have to be CStructArray i assume 10:52
otherwise we'd have to force all of the ecosystem to rewrite code to have CArray[Pointer[s]] instead
Skarsnik dunno, why a special type?
timotimo otherwise suddenly things would behave completely differently 10:53
Skarsnik nativecallable6, struct s {char *a;}; extern struct s foo[]; extern struct s* foo[]
nativecallable6 Skarsnik, gist.github.com/78af5d530089845296...5ea7128cee
Skarsnik nativecallable6, struct s {char *a;}; extern struct s foo[]; extern struct s* foo[];
nativecallable6 Skarsnik, gist.github.com/27bba45606df0dc0d2...79c429d6fb
Skarsnik nativecallable6, struct s {char *a;}; extern struct s foo[]; extern struct s* foo2[]; 10:54
nativecallable6 Skarsnik, class s is repr('CStruct') is export {ā¤ has Str $.a; # char* aā¤}ā¤our $foo is export = cglobal(LIB, "foo", CArray[s]);ā¤our $foo2 is export = cglobal(LIB, "foo2", CArray[s]);
Skarsnik right
timotimo yeah
we can't really have a transition period for switching both around, can we?
Skarsnik I think it could be worth to check if people use this form 10:55
I think a better question: is there a test that rely on this? 10:59
timotimo you remember testshellmem.p6? 11:01
Skarsnik github.com/rakudo/rakudo/blob/nom/...rays.t#L79 11:02
hm
not really ^^ 11:03
timotimo gist.github.com/Skarsnik/6adfba55f...e923112393 11:04
Skarsnik ohh this
well it show the virtual memory not the resident
timotimo oh, that's a thing. 11:05
anyway, if you run it more than just once, for example 5 times, you'll see it plateau after just a few runs
Skarsnik Yeah it grows up to 300Mb for me and stop growning 11:06
timotimo gist.github.com/timo/6a6550b8eac82...728e4ff3cb - check it
dogbert2 stmuk_, jnthn: dunno if it's related but the array allocation check was changed with commit github.com/MoarVM/MoarVM/commit/f9...1351ca6058 in order to fix an RT
timotimo 263 megs of maxrss for me 11:07
let's see how it goes without force_gc
Skarsnik I think jnthn explained that it was since proc:async create a thread it take virtualy like 8Mb of memory 11:08
for the thread
timotimo, if you have an updated l:p:s you can replace the mem call with like get-statm-human.perl
timotimo i do 11:09
wow, it really doesn't have to spawn these threads though 11:11
these threads should totally already be free for more work the moment new work is asked for 11:12
Skarsnik *need to start a game to free some ram* 11:14
timotimo puts some debug info into the ThreadPoolScheduler 11:21
Skarsnik rt.perl.org/Ticket/Display.html?id=131003 hm this leak I think, each pass make the process grow in memory 11:22
(aside from crashing after while)
maybe it's related 11:23
timotimo i'm still convinced that's a user error there
probably a problem in the binding 11:24
i think i found a mistake in the calculation for "do we need more threads" 11:28
i don't understand why asking the scheduler for its queue would always start a new thread unconditionally (except if the thread limit is already reached) 11:31
Skarsnik timotimo, dunno, I checked the binding lot of time and I respect what the C lib expect and the C lib handle its memory alone, you never have to free stuff it give you, you call a function when it's done and it do it 11:34
The only thing I don't trust from NC is cglobals/extern github.com/Skarsnik/perl6-gumbo/bl...er.pm6#L56 I hope this var is never messed by Moar 11:36
timotimo OK, then what causes the destroy function to be called?
Skarsnik I call it manually
timotimo and somehow you use the result again afterwards?
Skarsnik result? 11:37
timotimo something gets nativecall_refresh called on it
Skarsnik I mean the gist is : gumbo_parse(my_html); parse the tree and construct p6 XML tree; gumbo_destroy; 11:38
timotimo like a cstruct or something that was passed in some call
can you figure out what nativecall is invoked where it does the invalid read of size 8?
i think telemeh has something to print that out?
anyway, the thread run thingie now only spawns up to 4 threads 11:39
i hope this doesn't deadlock in spec tests or something
Skarsnik hm do I need to do something to run valgrind? 11:40
timotimo perl6-valgrind-m does it for you
telemeh is nonblocking, though. results will appear a bit after they have happened 11:41
and since valgrind serializes threads, you might wait for a long time before it outputs the thing
Skarsnik I hate this bug, because I try to reproduce on my 32 bit VM on a Moar where I printf every NC_invoke it does not crash xD 11:42
timotimo 32bit VMs are terrible, clearly 11:44
Skarsnik good changing the website fetched it crash faster 11:45
timotimo a whole bunch of tests were failing because ThreadPoolScheduler was outputting stuff on stderr ... 11:47
Skarsnik timotimo, btw I changed github.com/MoarVM/MoarVM/blob/mast...all.c#L761 libefence was crashing on this. I think there was a * too much x) 11:49
timotimo which * did you throw out? 11:51
Skarsnik I changed it to cptr = (void*)((uintptr_t)storage + (uintptr_t)repr_data->struct_offsets[i]); 11:52
the uintptr_t are here after reading a bit of www.securecoding.cert.org/confluen...to+pointer 11:53
(still running with efence) weird thing before changing this line I had some mmap error (that were uncatched since it did not crash?) but they disapeared 11:58
maybe I mess up this function x)
it's annoying, each iteration take more time that the previous one when running this in valgrind :( 12:00
timotimo mhm 12:01
Skarsnik and of course it does not crash as fast as without it x) 12:05
timotimo # got out: "start\nstart\nstart\nstart\nend\nend\nstart\nstart\nend\nend\nstart\nstart\nend\nend\nstart\nend\nend\nend\n" 12:06
# expected out: "start\nstart\nstart\nstart\nstart\nstart\nstart\nstart\nstart\nend\nend\nend\nend\nend\nend\nend\nend\nend\n"
Skarsnik the thread test? 12:07
timotimo yeah 12:08
Skarsnik I love how memory grow show in stuff like valgrind
{data => 640.672 kB, dirty => 0 kB, lib => 0 kB, resident => 557.464 kB, share => 21.804 kB, size => 695.312 kB, text => 2.808 kB}
41
{data => 649.928 kB, dirty => 0 kB, lib => 0 kB, resident => 563.308 kB, share => 21.808 kB, size => 704.568 kB, text => 2.808 kB}
42
{data => 650.760 kB, dirty => 0 kB, lib => 0 kB, resident => 568.696 kB, share => 21.808 kB, size => 705.400 kB, text => 2.808 kB}
timotimo i must have totally messed up the scheduler with my naive ideas
"only start a new thread if there's less than 1 idle thread" 12:11
is what i thought
before my change to one thing it was "start a new thread if there's enough free threads"
which i thought was wrong 12:12
but there were other operations that called "maybe_new_thread" and i'm pretty sure at least some of them are not useful in the slightest
the number of loads is not what i expected here
[Coke] I just did a git pull on rakudo, reconfig'd and moarvm built ok. was it a master-only issue? 12:14
(for the windows thing) 12:15
timotimo no clue :(
stmuk_ I was mistaken moar.exe builds .. the error is in NQP it seems
timotimo probably?
moar doesn't execute moar while it builds
and the error you're getting is one you can only get if you're executing moar bytecode 12:16
[Coke] reconfig's with master/master
timotimo i see what's wrong
Skarsnik timotimo, I just have a summarized leak report for valgrind. probably not useful? 12:17
==6682== definitely lost: 5,022,912 bytes in 131,826 blocks
==6682== indirectly lost: 27,800 bytes in 782 blocks
timotimo not useful, no. we need the invalid read
so my code makes the mistake of assuming $!loads is accurate enough to plan ahead
but there's a time span between "should i spawn a new thread to handle this task?" and "the thread has started up, grabbed the work, and increased the $!loads accordingly" 12:18
Skarsnik what could I give to valgrind to be more useful?
[Coke] c:\users\user\rakudo\nqp\moarvm\src\strings\parse_num.c(265) : warning C4715: 'parse_simple_number': not all control paths return a value
oops
c:\users\user\rakudo\nqp\moarvm\src\strings\parse_num.c(265) : warning C4715: 'parse_simple_number': not all control paths return a value
... this cut and past thing is proving problematic. :)
src\profiler\heapsnapshot.c(823): warning C4293: '<<': shift count negative or too big, undefined behavior 12:19
\o/
(small victories) 12:20
I was able to build with --gen-nqp=master and --gen-moar=master on win64 with strawberry perl and msvc.
er, s/build/config/
(running nmake now) 12:21
nmake test: the cpp nativecall tests failed, but that's it 12:24
timotimo cool i think i have a properer ThreadPoolScheduler pool size handling thingie now
jnthn: wanna review a change i made to the ThreadPoolScheduler? :) 12:32
i'm almost done with a spectest run
aha! spec tests green! 12:33
Skarsnik ^^ 12:34
stmuk_ I still get the array error with --gen-nqp=master and --gen-moar=master on strawberry perl
there was another report of this error with windows as well (which is why I tried it)
[Coke] stmuk_: what compiler?
(works fine with strawberry perl + msvc2017 here) 12:35
stmuk_ gcc version 4.9.2 (x86_64-posix-sjlj, built by strawberryperl.com project)
timotimo Skarsnik: check out the rakudo branch economic_thread_pool_scheduler 12:36
hm, it still spawns more threads than i'd hoped for 12:38
[Coke] can't even start a build with gcc, it seems. moar detection dies immediately 12:39
stmuk_ [Coke]: which gcc? 12:40
[Coke] whatever sperl installed; ... ah. had to get clean -xdf rakudo/nqp and moarvm to get rid of previous build cruft, apparently.
stmuk_ sperl installs different versions depending on its own version 12:41
timotimo aha, found my mistake
stmuk_ bbl 12:43
[Coke] gcc (x86_64-posix-seh, Built by strawberryperl.com project) 7.1.0 -- yup, there's the error. 12:44
stmuk_ I couldnt get appveyor builds working with sperl gcc > 7.something
maybe appveyor problem dunno 12:45
timotimo hmpf. my attempt fails the sleepsort one looks like 13:02
who would have thought: the thing i thought would be easy is actually difficult! 13:04
13:12 AlexDaniel joined
jnthn Scheduling *is* difficult :) 13:14
timotimo especially if you have to appease someone who wants sleepsort to work :D 13:15
timotimo deleted the branch 13:16
13:19 brrt joined
timotimo Skarsnik: i suggest you instantiate your own ThreadPoolScheduler and set its max threads to 2 or 3 :P 13:26
except you have to be careful because those are app-lifetime threads that won't be shut down when the TPS gets destroyed 13:27
13:40 brrt joined
Skarsnik timotimo, I don't have the invalid read size www.nyo.fr/~skarsnik/tmp/valgrind.txt 13:42
dogbert2 jnthn: didn't you fix RT #132042 the other day or am I dreaming? 13:47
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=132042
jnthn dogbert2: ugexe wrote a PR that probably does, I commented on it, maybe that feedback is already handled :) 13:48
dogbert2 aha, thx, just run the example a number of times and it seemed to work without a hitch 13:49
*ran
timotimo Skarsnik: that's something i've never seen before 13:54
dogbert2 timotimo: are you looking at the Gumbo bug? 13:55
timotimo not completely
i was looking through tabs i had kept open, one of them was the memory increase from the "run a shell command" thing 13:56
and i wanted to perhaps look closer at Log::Async
but i got sidetracked and lost
dogbert2 oh no 13:59
shouldn't we update libuv at some point? 14:08
brrt we should do so regularly, in fact 14:09
dogbert2 we're at 1.8 atm
timotimo yeah
we should
someone should :P
dogbert2 is it easy?
timotimo should be
dogbert2 I guess there's a place where we tell it which commit to use? 14:10
1.14.1 is the latest and greatest 14:11
timotimo you can cd into the folder it's in, pull and checkout something, then cd up and "git add" the folder as if it were a file
dogbert2 is that how it's usually done? 14:13
timotimo i think so 14:15
it's git submodules. they aren't intuitive until you learn what they are 14:16
dogbert2 brr, nothing for me then, at the very least I can check out the latest ver locally and run a spectest 14:18
dogbert2 spectesting with libuv 1.14.1 14:19
timotimo: didn't you fight with submodules before? 14:20
Skarsnik submodule are kinda annoying, since they are tied to a commit on the subrepo 14:21
You can't say 'follow master on this'
dogbert2 FWIW, the latest libuv doesn't upset the spectests one bit :) 14:24
timotimo i did a very bad thing with submodules
there was a folder in the repo that had stuff in it. i later replaced that folder with a submodule. git was *not* allowing people to just "git pull" that change because "hey there's already a folder here where i'm supposed to put this submodule file thingie" 14:25
dogbert2 the last libuv merge was done by Bart W. 14:26
could that be brrt
timotimo yes 14:27
dogbert2 can we entice him to do it again :)
timotimo oh, you don't have to learn much about submodules, you just need to do what i told you :P
dogbert2 but it should be done in a PR I guess 14:28
timotimo sure
dogbert2 hmm, I could give it a shot I guess 14:29
timotimo cool 14:30
dogbert2 timotimo: "pull and checkout something", do you mean checkout master, pull and then checkout the 1.14.1 commit ?
timotimo hm, right, it might be a better idea to fetch instead of pull 14:31
submodules tend to be in "detached head" state
Skarsnik it's probably the same? 14:32
since the git add will tie it to the specific commit
timotimo git pull might complain that you're not on a branch
dogbert2 so 'git checkout master' followed by 'git fetch' and then 'git checkout b0f9fb2a07a5e6'
timotimo with the fetch you don't need the first checkout, but i suppose that'd work 14:34
dogbert2 then I'll do it, PR coming 14:35
Geth MoarVM: dogbert17++ created pull request #682:
Bump libuv to ver. 1.14.1
14:55
dogbert2 brrr
stmuk_ timotimo: gist.github.com/stmuk/365b6cd98ed3...80f2369128 15:13
timotimo can you go into the set_elems frame and print the relevant locals and the result of that expression it uses? 15:14
ssize, CHAR_BIT, elem_addr_size
stmuk_ something like "break set_elems" and "info frame"? 15:23
timotimo no, just "thread " the right thread and then up up up up up until you're in set_elems 15:24
15:27 geekosaur joined
stmuk_ ok I'm in set_elems 15:28
timotimo you can only print one thing at a time, annoyingly 15:29
because of how the , operator in c works
stmuk_ github.com/MoarVM/MoarVM/blob/mast...ray.c#L455 15:34
I don't see ssize etc. 15:35
timotimo oh sorry
has to be set_size_internal
15:37 geekosaur joined
stmuk_ arggg I can't single step due to no line information 15:43
I just compiled moar with --debug .. is there anything else I need?
15:45 brrt joined
timotimo i tend to --debug=3, in order to have variables not say "value optimized out" all the damn time you should also --optimize=0 16:09
stmuk_ hmm I get full debug info on linux but not on windows .. must be the toolchain 16:41
timotimo we should be passing -g to the linker as well, that should keep debug symbols in the exe and dll files 16:42
Skarsnik can we have debug symbol in a separate file? 16:43
it's kinda handy in debien when you can just apt-get install lib-debug 16:44
to have the debug symbol
timotimo we can, yeah 16:46
16:56 zakharyas joined 17:07 TimToady joined 17:18 domidumont joined
Geth MoarVM: 1396733271 | (Nick Logan)++ (committed using GitHub Web editor) | src/io/io.c
Mark GC blocked when acquiring mutex

Fixes: `perl6 -e 'my $foo = 0; for ^50000 { start { say $foo++ } };'`
  irclog.perlgeek.de/perl6-dev/2017-...i_15127640
  rt.perl.org/Ticket/Display.html?id=132042
18:07
MoarVM: 1de91704d7 | (Nick Logan)++ (committed using GitHub Web editor) | src/io/io.c
MVMROOT remaining pointers to GC objects

All pointers to garbage collected objects accessed after a GC triggering function must be rooted.
MoarVM: 32322f3933 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/io/io.c
Merge pull request #679 from ugexe/patch-1

Mark GC blocked when acquiring mutex
MoarVM: d05764fb83 | Mario++ (committed using GitHub Web editor) | src/6model/reprs/P6bigint.h
MVM_IS_32BIT_INT(i) with explicit casts

see also
  github.com/MoarVM/MoarVM/issues/59...-326253445
18:08
MoarVM: 1e735a8998 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/6model/reprs/P6bigint.h
Merge pull request #671 from duke-m/patch-1

MVM_IS_32BIT_INT(i) with explicit casts
MoarVM/master: 5 commits pushed by MasterDuke17++, (Jonathan Worthington)++ 18:15
jnthn Well, at least a tad less behind on review now :) 18:23
dogbert2 jnthn: preparing an answer to your comment wrt github.com/MoarVM/MoarVM/pull/682 18:25
timotimo bleh, the mac builds aren't even starting for that pr 18:26
dogbert2 timotimo ?
timotimo it has been going for almost 4 hours now 18:27
8 x 3 minutes + 3 more minutes for covering
and lots of waiting for the mac builds to start
dogbert2 is it Travis causing trouble?
timotimo no clue
[Coke] need a mac build on something? 18:28
timotimo no 18:30
well, we could use a test on mac
[Coke] any branch in particular?
dogbert2 it's a scary change, things work for me but I only have a 32 bit Linux to play with 18:31
timotimo Pull Request #682 Bump libuv to ver. 1.14.1
in dogbert17/update-libuv
dogbert2 the more people who tests the PR the better :) 18:32
[Coke] trying perl Configure.pl --backends=moar --gen-nqp=master --gen-moar=dogbert17/update-libuv in a fresh rakudo.... 18:33
dogbert2 hides 18:34
[Coke] ah, PR. slightly more complicated, will report back.
dogbert2 timotimo: you there? 18:47
Skarsnik *3h of crashgumbo.p6 with efence, only 14 iterations* 18:59
[Coke] dogbert2: ... work busy, will get to this tonight. 19:09
dogbert2 [Coke]: thx, realised that my own testing had been bogus so I had to redo it, still works for me though 19:14
my mistake was that I checked out the latest libuv into the 3rdparty/libuv dir and then promptly did 'perl Configure.pl --debug --prefix=<installdir>' 19:16
I failed to realise that perl Configure.pl syncs all submodules so I was back to ver 1.8 :( 19:17
timotimo dogbert2: i was grocery shopping 19:23
dogbert2 timotimo: what do you think about my nOoB mistake above :) 19:25
perhaps it would have been better if you or Brrt did this 19:27
timotimo i would probably have been bitten by the same thing 19:28
weird. configure.pl didn't revert the version on my machine 19:29
19:31 zakharyas joined
dogbert2 if (-d '.git') { 19:32
print dots("Updating submodules");
my $msg = qx{git submodule sync --quiet && git submodule --quiet update --init 2>&1};
if ($? >> 8 == 0) { print "OK\n" }
else { softfail("git error: $msg") }
}
timotimo yeah, i know it did that
but the submodule stayed at the tag i checked out
dogbert2 I had to remove those line temporarily when testing 19:33
timotimo releasable6: status 19:34
releasable6 timotimo, Next release in 7 days and ā‰ˆ23 hours. 1 blocker. Changelog for this release was not started yet
timotimo, Details: gist.github.com/aaf4a7447d2e6200c5...aad93c82a2
19:50 domidumont joined 21:04 tartal joined
samcv i'm trying to get the collation data to concatenate in the creation of unicode.c but it's not working 21:35
MasterDuke samcv: btw, did you see the thing dogbert17_ found recently that bisected to one of the grapheme iterator commits? 21:38
samcv no
MasterDuke irclog.perlgeek.de/moarvm/2017-09-07#i_15131058 i think 21:39
timotimo i ran stresstest and it's all green except for t/spec/S17-procasync/bind-handles.t which hung (0 cpu) when run in the harness, but running it on its own does nothing to it 21:40
samcv $(CMD)$(CAT) src/strings/unicode_db.c src/strings/unicode_uca.c src/strings/unicode_ops.c > $@ $(NOERR)
i have this in Makefile. and i added to Makefile.in
but it doesn't seem to include src/strings/unicode_uca.c in the output file 21:41
not sure how to figure out why
if i do cat .... > src/strings/unicode.c with the three as listed in the flie it works but not when i run make
jnthn You re-configured? 21:42
samcv uh 21:43
jnthn Oh, you said you have it in Makefile, so yes
samcv i changed Makefile.in
and then ran configure again
jnthn Right, that'd be sufficient
samcv hm
jnthn (I figured you'd have done so, but I know I've forgotten before and had a confused moment... :))
21:44 AlexDaniel joined
samcv gist.github.com/9b922187b33eee81cd...2933d3e8cf here's the Makefile.in 21:44
jnthn Should there be a space between $(CMD) and $(CAT) ?
samcv feels like it should be right. but it doesn't concat it
CMD isn't defined to antything on linux
i think 21:45
jnthn Yeah
I note the other places that use $(CAT) don't put $(CMD) before it
samcv hmm lemme remove
jnthn Curiously, none of the other examples of $(CAT) in that Makefile invoke it with multiple files 21:46
samcv works
if i remove $(CMD)
jnthn hm, curious 21:47
samcv hmm i think i maybe found the issue 21:51
gi->start isn't initialized in MVM_string_gi_init
if it's a flat string
or could be unrelated
ah no that makes perfect sence 21:53
i'm not seeing the error
MasterDuke might be a 32bit thing 21:54
samcv no i get it too 21:55
move_ti for grapheme iterater doesn't set gi->start for flat strings
at least it didn't cause an issue since flat strings don't have to move to future strands so it was alright 21:59
Geth MoarVM: 0441e075b4 | (Samantha McVey)++ | src/strings/iter.h
Ensure gi->start is set to 0 for flat strings in MVM_string_gi_init

This was causing a valgrind warning about an uninitialized value being used. This did not cause any actual bug since start is an unsigned integer and we don't actually have to move to other strands for flat strings.
22:05
samcv dogbert17_, fixed :)
jnthn, putting a space between $(CMD) and $(CAT) also fixes the issue 22:07
jnthn ah, nice 22:11
Well, happy it works :)
22:22 dogbert2 joined
dogbert2 samcv++ 22:22
samcv cool. i got both the nodes and the collation array data using my bitfield packing module to ensure both are packed properly 22:27
not sure if it matters for either after the change but at least i know in the future it can change in case more are added