jnthn I've just cut stage mast in half. 00:00
Uh, to less than half.
diakopter er, wow.
such cut
jnthn need spectest
desire verify
00:01 woolfy joined
diakopter jnthn: what was your big fix 00:05
jnthn The funny thing is it's a constant factor fix rather than an algorithmic one... 00:07
Patching coming up
dalek arVM: e072c4c | jnthn++ | src/6model/ (6 files):
Optimize root objects list in SC.

This doesn't make things better from an algorithmic point of view, but looking up objects in SCs - and in particular, searching the SC to find the index of an object - is a very common operation. This makes all SC object operations a good bit faster, and easier for the compiler to do inlining on, which is a very notable win.		 00:09
jnthn The spectests don't hit the serialize path that was terribly slow, so it's only several more seconds off that. 00:10
diakopter oh no :) 00:11
jnthn Down to 329s.
diakopter wait, why couldn't you mark the index on the obj itself
jnthn Bloats all the obj...
Most objects aren't ever in an SC
diakopter but if it was only like 8 bits from the bitflag field 00:12
jnthn But it's not that, it's more
uh
I mean, there's more than 256 things
diakopter I figured ?) 00:13
jnthn But yeah, you could << 8 and cap
diakopter oh, then linear search just the multiples :)
jnthn No, that's cache-horrid. I meant you could know you could skip that number << 8 entries before starting searching.
diakopter oh 00:14
jnthn I need to re-profile to see where the costs are now
diakopter well how about this - since they're always deserialized into gen2, do a hashtable on their memory address :D :D
(and then expire the hashtable if they move..) 00:15
jnthn Yeah, a hash table approach could be wise.
diakopter how much time was it spending in sc index lookup previously 00:16
(did the profiler indicate that?)
jnthn Around 15-20% of total time for compiling CORE.setting 00:17
diakopter I seem to recall being the one to code that linear search 00:18
.oO( oh, this won't matter, right, except when compiling... )
jnthn Well, it's linear search elsewhere too 00:19
Now it's still worth improving, but much smaller
Like, way down the list.
26% of time goes on memory management, with 25% of that on the collection side. 00:20
Which is kinda what you'd expect given we do bump-the-pointer allocation.
4% of the time is on the walk through the old nursery to gc_free things that need it. That's possible to push off to another thread. 00:22
diakopter yup
I noted that in a comment in the code in the gcorch branch 00:23
(and then you noted it in master)
I think
jnthn Well, and I knew we could at the point I designed Moar ;) 00:24
try_get_slot is very hot.
timotimo: How far did your hintfor work get?
If this profile is right, which I've little reason to doubt, we spend over 5% of our time looking up dynvars. 00:25
$*foo and the like
4.6% on string comparison 00:26
diakopter eww 00:27
how many of those successes?
jnthn Most...
Oh, it's 'cus attribute lookup
Once we have hintfor a bunch of those go away.
And when we've a specializer another load to 00:28
*do
diakopter jnthn: I think timotimo pasted a link to a diff or something 00:32
[Coke] ponders hacking on sixperl or working on his sixperl presentation. 00:33
[Coke] does the former and calls it "research"
diakopter I'm good at that kind of research too
jnthn Um, wow 00:36
diakopter so scare 00:37
jnthn Is it me, or is the MVMStaticFrame marked refs_frames for no reason...
diakopter sounds like git blame diakopter
jnthn coulda been me 00:38
And it could be that it once did
diakopter wait, why doesn't it
jnthn But we spend 6.8% of time adding inter-generational roots to the worklist 00:39
[Coke] m: class A { has $.a = 0 }; for ^2 { my $a = .a given A.new; say $a; } # RT 121049
camelia rakudo-moar 07c483: OUTPUTĀ«0ā¤0ā¤Ā»
jnthn But just under half of that is calling gc_mark on MVMStaticFrame instances.
[Coke] ^^ I cannot duplicate raydiak++'s bug report.
diakopter oh
jnthn [Coke]: That's 'cus I already fixed it. 00:40
[Coke] jnthn: but the ticket is still open. :P
jnthn :P
[Coke] Is there a test?
... if not, that's fine. 00:41
jnthn No, no test
Oh, unless raydiak already wrote one
We talked about where to put it but I don't recall seeing it going in 00:42
(S04-statement-modifiers/given.t)
[Coke] danke.
00:43 benabik joined
jnthn Well, seems s/1/0/ shaved another couple of seconds off CORE.setting 00:43
lee__ benabik: i'm getting a fun new libmoar error when building rakudo on OS X :P do i need to pull another patch for it to work? 00:44
benabik lee__: AFAIK, Rakudo master builds on OS X. :-/
lee__ darn, ok
benabik What was the error? 00:45
lee__ here is the error: gist.github.com/leedo/ba22b9c6ed7220ef21ca
benabik Hmmmmmmm.... 00:46
lee__ i actually get the same error when using gcc 00:47
though i should double check that
benabik Maybe I should change it from -l to just a path to the library. 00:48
I thought rpath would make it look in the right place.
It probably works for me because it's installed in /usr/local/lib and that's in my linker search path already.
lee__ ah, makes sense
00:48 Util joined
lee__ you'd think rpath would take care of that 00:49
benabik rpath is part of the runtime search. I guess ld uses it's own at link time.
dalek arVM: 370f3cf | jnthn++ | src/6model/reprs/MVMStaticFrame.c:
MVMStaticFrame does not directly ref an MVMFrame.

Thus refs_frames can be cleared. It may have referenced them at some point in the past, but it doesn't now. Since MVMStaticFrame was the most costly of the refs_frames things to mark, and there were a lot of them, this makes the intergenerational roots a bunch cheaper to handle.		 00:51
jnthn Spectest time from 329s => 321s with that. 00:52
[Coke] ~>
diakopter XD 00:53
benabik Rakudo PR #244 should fix it. 00:54
jnthn benabik: merged 00:55
Thanks
benabik jnthn++
lee__: Try again. :-D
lee__ benabik: nice! trying, after i revert my bad attempt 00:56
benabik: missing -l ? 00:57
hmm or not 00:58
benabik lee__: Shouldn't need it when giving a path.
-lmoar => ...../libmoar.dylib 00:59
01:00 lizmat joined
lee__ ah i see, a new error. i wish i could be more help 01:00
gist.github.com/leedo/329cca50dc20f8a70b35
actually that is probably a side effect of my hacking 01:01
benabik Yeah, that seems odd.
I dunno where the dynext/ comes from.
Actually, no. I don't know where the rest of the path comes from. 01:02
It should just be dynext/libperl6_ops_moar.dylib
lee__ nuking and trying again 01:04
01:04 jnap joined
lee__ benabik: success! 01:19
thank you!
benabik \o/ 01:20
I'm glad. I realized I didn't reconfigure before testing, which is silly to do with build changes. O.o
I mean, it was Obviously Correct (tm), but... 01:21
jnthn benabik++ lee__++
02:02 benabik joined 02:37 cognominal joined 04:29 jnap joined
diakopter the LCTG for moar is taking significantly longer today 04:56
er.
LTCG
05:30 jnap joined 06:18 FROGGS joined 06:31 jnap joined 07:14 FROGGS joined
FROGGS a p-spectest shows a similar maxresident again... 2606.11user 163.62system 14:14.56elapsed 324%CPU (0avgtext+0avgdata 769168maxresident)k 07:15
MoarVM++ :o)
07:32 jnap joined
timotimo jnthn: i implemented hintfor and that's already in the stage0; my usage of it was still trying to "shoehorn" it into getattr and bindattr ops compilation, rather than the way you suggested it 07:53
it does seem to correctly get hints, though
jnthn: could the fix for the MVMStaticFrame thingie cause the algorithmic improvement in the branch you did the other day to work? 07:54
08:09 FROGGS joined 08:30 odc joined 08:32 jnap joined
timotimo oops, i was wrong 08:33
i added nqp::hintfor in a branch, not in master :( 08:34
and then i did an unnecessary stage0 update
d'oh
what an expensive blunder
FROGGS gah! 08:36
:P
FROGGS .oO( you are an insensitive clod )
timotimo here we go 08:38
and now for making bindattr and getattr use it, too
ah, hm, i need to make another version of getattr/bindattr for when the attribute name isn't constant :\ 08:51
dalek arVM: a0a4666 | diakopter++ | src/6model/6model.c:
type in no such method error.. I think
 09:27
09:33 jnap joined
nwc10 jnthn: once MoarVM handles things like native ints better, will that just speed up benchmarks that test this, or are they used extensively in NQP and the setting? 10:22
jnthn nwc10: The problem wasn't/isn't actually at MoarVM level 10:23
nwc10 oh, OK.
jnthn nwc10: NQP already does pretty well with them, afaik
nwc10 pretend I asked it on #perl6 then :-)
jnthn nwc10: The problem was that Rakudo didn't inline operators.
Now it does, but the inline comes out badly 10:24
I think the latter is an all-backends issue though
diakopter: You can't assume every meta-object is a KnowHOW. a0a4666 is gonna SEGV...
diakopter oh 10:26
:(
add a check for that repr?
oh, well, nm
dalek arVM: d7f4857 | diakopter++ | src/6model/6model.c:
gah
 10:27
jnthn Well, I was gonna go for a better fix than that, and have one that lets us actually get the invocant...
...in some HLL code, so we can construct a better typed exception.
diakopter oh
10:34 jnap joined
diakopter jnthn: did you fix the callsite signature dupe detection? :) 10:35
jnthn timotimo: Measurable change for me too with getattr/bindattr changes so far...
diakopter: Yes, it shaved tens of kilobytes off CORE.setting.moarvm.
diakopter oh :) 10:36
I wonder what I borked
jnthn I can't remember how I patched it...
git log --grep=callsite -p
:P
I was gonna say that getattr/binattr tweaks so far made no spectest difference, but then I noticed we ran 7 extra tests in about the same time :) 11:05
FROGGS that sounds like >99% to me 11:06
diakopter jnthn: I'm getting this when building with that: gist.github.com/diakopter/70d38c838f9633751983 11:10
jnthn my $obj_mast := $qastcomp.as_mast( $op[0] ); 11:13
my $type_mast := $qastcomp.as_mast( $op[1] ); my $obj_mast := $qastcomp.as_mast( $op[0] );
oops
diakopter oops :)
jnthn On those two I'd :want($MVM_reg_obj)
Just to be sure
diakopter ok
same error 11:14
nqp builds fast as lightning, btw 11:15
jnthn Well, sub-minute anyway...
Yeah, I don't immediately see why the error...
It's gotta be some silly though 11:16
'cus here's the line it blows up on: 11:17
nqp::bindattr($der, NQPRoutine, '$!dispatchees', nqp::clone($!dispatchees));
The the value there is an array
diakopter hm 11:18
missing decont on something/ 11:19
?
value maybe?
er, cont?
jnthn No, only class handle should be deconted and it is being 11:21
diakopter when I've seen that before it turned out to be a bug in another place in the compiler 11:23
reusing a register when it shouldn't have been
(such as the only one we release that could've been passed - the name str expr) 11:24
jnthn oooh
my $name_mast := $qastcomp.as_mast( :want($MVM_reg_str), $op[2] );
push_op(@ins, $namedop, $obj_mast.result_reg, $type_mast.result_reg,
$name_mast.result_reg, $val_mast.result_reg);
Forget to push_ilist(@ops, $name_mast); here
diakopter hee
good eye 11:25
timotimo: since you understood what the qast-generating version was doing, you should be able to see how it translates to the mast emission version
jnthn: erm 11:27
same error?
jnthn: oh, it needs box? 11:28
jnthn: oh, you're not going to believe this 11:30
jnthn Is there another bug too? :)
tadzik it made rakudo faster than perl 5 11:33
diakopter faster than go 11:34
tadzik come on, go is slow. It's like chess
diakopter I LIKE MAGLEV
jnthn Anybody for > 100 MB off CORE.setting compile memory use? :)
tadzik :o
11:35 jnap joined
diakopter jnthn: u fixed? 11:35
jnthn diakopter: Your patch? No, I did something else :) 11:37
diakopter: You want me to look at it, or you still working on it?
diakopter yeah I'll push and break the build :)
bbiab 11:40
dalek arVM: 2faf743 | jnthn++ | 3rdparty/uthash.h:
Reduce initial count of hash buckets from 32 to 8.

Most of the hashes we build are rather small. Changing this produces no noticable time difference to Rakudo CORE.setting build or spectest time, however does appear to shave 100MB or so off CORE.setting build memory use.		 11:41
jnthn diakopter: a...branch may have been better :P 11:42
dalek arVM: a85a25f | jnthn++ | src/6model/reprs/P6opaque.c:
Don't explode on uncomposed in hintfor.

It simply means we're not in a position to optimize.		 12:00
diakopter jnthn: any ideas why we're stuck with -O1 on linux? 12:12
tadzik to make windows look good :P
jnthn diakopter: No
diakopter: It needs some detective work to figure out which optimization(s) -O2 enables break us 12:13
FROGGS -O2 and -O3 explodes when compiling rakudo's BOOTSTRAP.pm 12:16
12:22 lizmat joined
jnthn We need 7s more off CORE.setting compilation time before it's within a minute for me. 12:25
nwc10 on your desktop? 12:27
jnthn Yes. 12:33
Though my current laptop is no slouch either.
It's almost competitive on CORE.setting build. It isn't on spectest 'cus it's only got 2p/4v cores, rather than 4p/8v on my desktop
One of my jobs for Feb: big MoarVM IO overhaul. 12:35
12:35 jnap joined
jnthn I'm really not happy with it. 12:36
No discredit to anybody who worked on it; I'm happy the amount that works does so.
But...the design seems to mean we're doing checks for things all over rather than figuring things once and then having a vtable-y thing that just takes us off to the right place. 12:37
And it's gonna get worse still once we add in async IO stuff.
nwc10 please don't steal from the Perl 5 PerlIO vtable 12:38
do ask leont what the Perl 5 should have had :-) 12:39
jnthn nwc10: Well, I wasn't really thinking of Perl 5 at all (I've no idea how its IO looks), but more replicating the approach I took on JVM 12:42
nwc10 aha
timotimo could it be that concat and trim_string suffer from MVM_string_flatten?
jnthn timotimo: Well, the flattening is always going on to avoid tickling bugs. 12:43
timotimo right.
i feared that :)
jnthn I'm hoping diakopter++ may get tuits to look at that stuff at some point. 12:44
timotimo also, i think the most common case for concat will be that our LHS has multiple strands and our RHS will have one
jnthn Yeah
timotimo but since the strings will always be flattened after concatenation, we'll end up with only one strand on either side for now
JimmyZ hopes to see the NFG branch 12:45
timotimo jnthn: if the flatten function resets the body.flags from MVM_STRING_TYPE_ROPE to MVM_STRING_TYPE_INT64, shouldn't it free the strands array? 12:48
or does it continue using the rope stuff?
jnthn timotimo: I'm really not sure without looking further into it
timotimo: At first thought, sounds like yes 12:49
timotimo i'll just try adding it and see if the spectests fail :P
jnthn Would it mean we never free it later?
timotimo maybe. let me check. 12:50
P6str doesn't have a gc_free or gc_cleanup 12:51
am i looking at the wrong place? 12:52
jnthn Wrong place
MVMString
timotimo OK
gc_free never touches the strands 12:53
so it would appear that this leaks
that doesn't explain why the complexity seems to be so bad, but ... whatevs. 12:54
timotimo runs spectests 12:56
hm, yeah, how do i tell if the failures are my fault or are currently normal? %) 12:57
jnthn Compare it to before you started :) 12:58
timotimo i'll have to generate that data first :| 12:59
luckily the spectest run is pretty darn fast
833.71user 37.61system 4:01.45elapsed 360%CPU (0avgtext+0avgdata 280704maxresident)k 13:00
jnthn Is that 280 with my recent patch for initial hash size, ooc? 13:01
timotimo it might be. let me check
yes, it is
4 minutes for a full spectest run is really good 13:02
moritz timotimo: on how many cores?
timotimo 4 13:03
i *think* 4 physical, 4 virtual
how do i find out?
jnthn Do you have hyperthreading enabled?
That's what gives you the virtual ones.
timotimo yeah, how do i find that out?
cpu cores: 4
jnthn Not sure on your OS :)
tadzik htop or /proc/cpuinfo gives you virtual ones 13:04
timotimo yeah, 4 of those
tadzik is that i5?
timotimo model name: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
tadzik I _think_ all i5s are 2 physical, 4 virtual
timotimo OK
i have an i7 in my laptop, but it's 2p, 4v, too 13:05
i seem to have made new b0rks with my free call in MVM_string_flatten
tadzik ok, wrong. It's 4 physical, and 4 HW threads 13:06
timotimo i'll let someone who knows the string implementation inside-out try it out
tadzik according to wikipedia
jnthn ah...then it probably ain't right...
tadzik and compared to my 3320M, which is 2/4
13:16 FROGGS joined
FROGGS timotimo: in case you have a core i5, you have two physical plus ht 13:16
timotimo fair enough 13:36
13:36 jnap joined
timotimo calloc allocates on the stack, doesn't it? 13:36
or was that alloca? 13:37
right, it's alloca
there should be some free for the calloc that creates the strands, i think.
FROGGS .oO( Living a vid alloca? )
JimmyZ calloc is malloc&&memset zeror 13:38
timotimo it's still amazing that moarvm takes 100 megs of rss for just "say 1"
JimmyZ *zero
timotimo but parrot takes 2.5x that
so ... yay i guess?
jnthn Probably lower than JVM too :) 13:42
timotimo yes, the jvm has another 200 megabytes more 13:43
jnthn o.O
timotimo 0.19user 0.04system 0:00.24elapsed 98%CPU (0avgtext+0avgdata 104036maxresident)k 13:44
0.28user 0.08system 0:00.39elapsed 92%CPU (0avgtext+0avgdata 247232maxresident)k
8.38user 0.20system 0:04.77elapsed 179%CPU (0avgtext+0avgdata 429280maxresident)k
i wonder how low we're going to get 13:52
lazy loading of the setting, for example, would probably be an amazing win to get the minimum memory usage down
if all you want to do is say 1, you don't need bags/mixes/complex/... 13:58
moritz or sockets 13:59
FROGGS that is an interesting idea... 14:00
timotimo that's an old idea :)
nwc10 stop thinking fun thoughts, and get back to the grindstone of making tests pass! :-) 14:01
timotimo what do we need to keep the .moarvm files around for? 14:03
i mean, mmapped in memory?
big parts of it are deserialized into in-memory structures after loading it
so maybe only the rest (bytecode i guess) is actually needed? 14:04
jnthn timotimo: Correct 14:09
timotimo: That's a plausible optimization
timotimo are the two pieces cleanly separated?
jnthn Well, sort of
timotimo hehe.
jnthn I mean, the bytecode is all in one big chunk
timotimo i think i can mmap /dev/zero over it and linux will DTRT 14:10
jnthn I think you need to keep annotations mapped in too
timotimo otherwise, memadvise can set an "unused" flag, but i don't think it'll get rid of the memory ASAP
are the annotations in one chunk, too?
at least we open the bytecode file readonly, so the kernel *should* share it between processes already 14:11
jnthn Yes, they're in another chunk. Maybe not right next to the bytecode one... 14:12
timotimo that'd be okay
nwc10 If I follow the munmap() man page correctly, you can unmap areas of memory (ie not the entire file) 14:13
so if that's right, you don't need to madvise(), and you don't need to map in zero
jnthn nwc10: That sounds just what we need.
Yeah, mapping in zero feels like a hack to me...
timotimo oh, that seems nice
nwc10 so, try to have the file ordered with things to keep contiguous
timotimo we'll only ever waste one page, i think
well, on either side
nwc10 does the deserialise happen before any bytecode is executed?
yes, that's what I thought too. At most 1 page each side 14:14
timotimo how big are pages? like 4 kbytes?
nwc10 I believe nearly everywhere, yes 14:15
jnthn nwc10: No
timotimo that doesn't sound too bad, though
nwc10 OK. No :-)
jnthn But we get chance to do the munmap afterwards
nwc10 oh, no, to deserialise
not to 4k 14:16
threads. What IRC lacks
jnthn Right :)
timotimo yes, 4096 says getpagesize on my machine
since the bytecode files are multiple megabytes big, that ought to be a small loss if the bytecode and annotations are both chunks and the rest around and between can be discarded 14:17
nwc10 I believe that this is how ELF lays out files. Sparse files, rounded to some size, which might be 32K 14:18
but I haven't checked that
timotimo we can surely embed some padding
14:27 jnap joined
timotimo a friend just pointed out that mmapped files don't count for the rss, only for "shared" memory 14:35
nwc10 oh. But I think that it's still a good idea 14:36
timotimo yeah 14:37
jnthn is working on making our code-gen less awfuls
timotimo jnthn: is there something i can do to implement nqp::locallifetime on moar? 14:38
i'm not sure if we use it anywhere besides $foo++ and friends, though
oh, in sinking we use it
ah, in many places actually 14:39
jnthn And ACCEPTS... 14:41
14:45 camelia joined 14:47 benabik joined
timotimo any mechanism in mast that would allow me to signal locallifetime to the mast compiler? 14:57
jnthn timotimo: Well, see how QAST::Stmt does it... 14:58
timotimo will do 15:00
have to finish my commute first
why exactly can't we just use stmt instead of locallifetime? 15:04
other than that i should be able to make that work somehow 15:05
jnthn Because we want to contorl which locals are killed
timotimo just need to find out what to put into the QAST tree so that the mast compiler knows that there even was ablocallifetime op 15:06
oh, is there an argument to locallifetime?
15:12 FROGGS[mobile] joined
jnthn timotimo: Yes, it's AST and then names of locals 15:23
timotimo fair enough 15:24
15:25 jnap1 joined
jnthn timotimo: I think you need to replicate the force-freeing that QAST::Stmt does, anyways 15:25
timotimo: Just that you know the names to force-free
timotimo yeah 15:26
jnthn Sadly, make release doesn't work at all well on Windows and I dunno what to do about it :) 15:31
nwc10 delegate! 15:32
moritz or write a better 'make release' target :-)
jnthn moritz: Yes, that's the "not sure how to do better" bit :)
moritz install tar and gzip? 15:33
jnthn Doesn't even get that far :) 15:34
[ -n "2014.01" ] || ( echo "\nTry 'make release VERSION=yyyy.mm'\n\n"; exit 1 )
'[' is not recognized as an internal or external command,
operable program or batch file.
timotimo %)
does windows have "test" instead?
moritz
.oO( install [ )
and echo
benabik And ( ? ;-)
jnthn moritz: make release doesn't seem to include the sub-modules in the .tar, anyway 15:35
nwc10 replace the shell with Perl (5, that is) and become portable
jnthn moritz: Was the intention that it would?
moritz jnthn: it did for me 15:36
jnthn moritz: The directories are there but empty
3rdparty/dyncall is empty, for example 15:37
moritz mlenz@mlenz-workstation:~/p6/MoarVM$ tar tzf MoarVM-2014.01|grep 3rdparty|wc -l
255
that's not just directories
maybe different git version?
huh, dyncall/ is empty
but libatomic_ops isn't
wtf?
jnthn libatomic_ops isn't a submodule, though?
No, it ain't... 15:38
moritz what about libtommath?
that's included in the tarball also
jnthn No
Only linenoise, libuv and dyncall are submodules
moritz ok, so I need a betters solution :( 15:39
probably the git ls-files hack from nqp/
benabik I don't know that ls-files follows submodule boundaries. 15:40
*crosses.
moritz it doesn't :(
wonder how nqp does it, then
benabik does nqp have submods?
moritz oh, probably doesn't
jnthn NQP doesn't seem to use submodules.
moritz I assumed that dyncall was one, but it isn't
maybe I'll have a clever idea while taking the subway home 15:41
jnthn Yeah, docs I@m reading seem to suggest submodules aren't included in by git archive
benabik You could do something like `git submodule foreach 'git ls-files | sed -e "s/^/$path/"'` 15:43
Oops, the /s in the path get interpreted by sed. 15:44
git submodules need to suck less.
moritz aye; they mostly suck by being ignored by most git commands right now 15:46
benabik notes that git.git uses subtree merges instead.
timotimo is that like submodules, but without having a separate repository for the submodule's object files? 15:47
benabik Kinda-sorta.
moritz you basically import a whole repo and their history into your own
benabik Right.
timotimo i really love the simplicity of gits object database 15:48
moritz but shove the files into a subdir
benabik The subtree merge is basically "do a merge as-if the 2nd parent was at /path the whole time"
moritz hasn't actually used subtree yet
benabik I think git.git/git-gui used to be a s 15:49
*submodule
Oh, no, it just used to be part of the mainline. :-/ 15:50
Submodules are something of a clever hack that got accepted but never quite integrated. 15:51
jnthn timotimo: Just cut natives time in half again for a loop up to a million... 15:52
timotimo wow.
jnthn timotimo: So if you didn't start benchmarking yet...
Yeah, the code-gen is almost decnt now
timotimo hits ctrl-c
jnthn decunt
timotimo no i didn't start yet!
jnthn decent
dammit
:)
timotimo decont? :)
jnthn :P
I need to spectest this latest change 15:53
timotimo sure, go ahead
jnthn I'm not completely happy with it yet, but it's a lot better
diakopter mourn 16:03
moarn
timotimo can we somehow do a cheap indexing step over the SC and then only deserialize the stuff we find ourselves actually needing at the first moment we need them? 16:05
jnthn timotimo: Not easily
timotimo OK
hm. does deserialization parallelise at all?
jnthn Not yet
timotimo i mean in theory :)
jnthn Oh...
It may well be able to
timotimo that sounds nice 16:06
jnthn Depends whihc phase. :)
But probably it can.
timotimo could shave off another third of startup perhaps
jnthn Yeah, but question is how much we can shave off by spending some time looking at what makes the serialization blob large and seeing if we can make it smaller
timotimo that'd be something, too 16:07
what do we think of variable-sized ints a la google protocol buffers and Sereal?
jnthn Is it a similar scheme the CLR uses, using higher bits to indicate how much should be read?
timotimo i think so 16:08
that could very well quarter a big portion of our ints
and make a small-ish portion of our ints a bit bigger
jnthn aye
timotimo though, if we could find a way to express "all the leading bits are 1" or rather "this number is negative", we should be able to save even more 16:09
jnthn True
It should be quite easy to try, anyway
timotimo moarvm files already carry their serialization version info with them, aye? 16:10
jnthn Right
Well
The serialization blob starts with it
timotimo sounds good. i shall dig in :)
This encoding uses the high bit of each byte to signal there is another byte worth of data coming, and the last byte always having the high bit off. ā† sereal's varint, google's varint128 16:13
er, what file exactly should i be looking at? 16:14
bytecode.c looks the place 16:16
at least for loading 16:17
jnthn timotimo: No, src/6model/serialization.c 16:18
timotimo ah, thanks
jnthn Cool, I just got a 14s nqp make test (non-parallel) 16:19
timotimo wowza. 16:21
dalek arVM: 9680f23 | jnthn++ | src/strings/ops.c:
Two strings at the same location are equal.

Duh.		 16:22
timotimo jnthn: that gives us how big a speed improvement? :)
jnthn Not huge, but I saw stage mast dip just below 20s for the first time ever here 16:24
FROGGS \o/ 16:25
jnthn Will know if there's a spectest time improvement soon
FROGGS .oO( sooner? ) 16:27
jnthn Few more seconds off
timotimo ghlad to hear it 16:32
the benchmarks have finished :)
jnthn ooh 16:33
link?
timotimo soon 16:34
timotimo tries to get access to his webspace
i'll just host it on my desktop 16:40
diakopter jnthn: I swear it compared memory addresses before 16:49
jnthn diakopter: Dunno. It sure does now :) 16:50
We spend a LOT of our time doing hash lookups.
And we don't cache the computed hash value anywhere...
I'm wondering if we maybe should start doing so. 16:51
diakopter er I thought we did 16:52
jnthn Nope
We re-compute it every time.
diakopter weird...
timotimo jnthn: you do have ipv6, right? 16:54
jnthn Also, looking in MVM_string_flatten
buffer = calloc(sizeof(MVMCodepoint32), sgraphs);
for (; position < sgraphs; position++) {
/* XXX make this use the iterator */
buffer[position] = MVM_string_get_codepoint_at_nocheck(tc, s, position);
Is there any reason to use calloc there?
It looks to me like we fill the thing 16:55
jnthn tries changing it to malloc
diakopter I'd need a url to expediently find the source 16:56
jnthn Don't worry, I'm fairly happy I analyzed it right
diakopter oki 16:59
moritz github.com/meitar/git-archive-all....ive-all.sh that would solve the git archive + submodules problem 17:02
gist.github.com/moritz/8562794 # hacky shell script for creating a MoarVM tarball 17:11
jnthn moritz++ 17:13
Feel free to commit it in tools
timotimo jnthn: the concat benchmark spends 50% of the complete time doing MVM_string_get_codepoint_at_nocheck 17:28
i think that's from flattening the strings
it could be that using an iterator woul dhelp 17:29
jnthn Yeah
OK, I think I got working caching of hash vlaues on strings. 17:33
*values
diakopter it's not from flattening the strings 17:36
masak TIL 'exec >MANIFEST'. cute. moritz++
tadzik cat: MANIFEST: input file is output file
ain't that funny :P
diakopter in fact, if you can assume they're all flat, you can write a much more efficent comparator
*efficient 17:37
but any decent optimizer should inline that call anyway
er, sry, sufficiently smart optimizer :)
17:38 jnap joined
timotimo diakopter: what is flattening? 17:39
it would seem if the string types of the individual strands are compatible, we can just memcpy the pieces together
diakopter well all the strin goperations that allocate a new string as the result try to conserve copying of codepoint data, so it has its own substring reference system 17:40
timotimo yeah, but at the moment a concat calls the flattener
diakopter right, because of some gremlins
timotimo which does a loop through the individual codepoints, going through the get_codepoint_at_nocheck thingie
if you can get rid of that, the benchmarks ought to improve drastically
have at it! :)
diakopter (which jnthn hopes I'll have time to address soon)
timotimo fine :) 17:41
diakopter the flattener has to do that, except the note I mentioned about using the iterator functor
(the XXX)
since it would be traversing the tree otherwise anyway 17:42
timotimo yeah, but the concatenator doesn't have to flatten all the things
diakopter it does, because of gremlins
(currently)
gremlins with the tree-strings system, which originated (in spirit) from this post: www.nntp.perl.org/group/perl.perl6....g3236.html 17:44
timotimo it should of course flatten parts
if we add one "x" per concat, we'll get a quite horrible rope
diakopter yes, see the todo notes at the top of ops.c
timotimo i don't know at what length a rope breaks even
so it should perhpas be twice that number
diakopter the todo notes discuss that 17:46
(needing to discover those thresholds)
iirc
timotimo fair enough
dalek arVM: 01b7dfa | jnthn++ | src/strings/ops.c:
Don't calloc when malloc will do fine.
 17:51
arVM: dcc4340 | jnthn++ | / (3 files):
Cache hash values of strings.
diakopter it's actually not similar to ropes, so I don't call it that
[anymore]
jnthn Down to a 63s CORE.setting build here, and my spectest run just completed in 304s. 17:52
FROGGS 304s is not much 18:01
jnthn And yet it leaves much room for improvement :) 18:03
FROGGS yes :o) 18:04
FROGGS .oO( spectest in less time than getting a coffee )
diakopter jnthn: if you can make all the windows system .dlls late-bound dynamically linked instead of always loaded at every launch by the crt (or else fix the static compile) that would save 22ms off the startup, according what I can see 18:06
jnthn diakopter: I'm not sure how to do that... 18:07
diakopter fix the static compile? :)
jnthn Well, we can't jsut do --static all the time, if that's what you mean... 18:08
At least, I don't think so...
diakopter I thought you were saying it's only rakudo that couldn't handle it 18:12
(so far)
b/c I thought we built both versions 18:13
er, I think not_gerd set it up to build static and dynamic in the same makefile eventually
so that we could do such a thing eventually
jnthn Yeah, but Rakudo relies on the dynamic... 18:15
diakopter but it doesn't have to if we make it build static too 18:16
just sayin' we could reduce the startup time to within p5 levels or a binary order of magnitude :) 18:17
[Coke] arglebargle. the scripts I have for building rakudo for the daily runs work fine offline... except for moar, which is doing a git pull somewhere and failing. 18:20
(during config, dealing with 3rdparty stuff)
[Coke] wonders if he can trick it to not go online, just like the nqp/moar fetches. 18:21
nwc10 gah, firefox stupid
I *can* SOCKS proxy over IPv4 to a machine with IPv6 18:22
(Actually this machine I'd IRCing from)
but I have to use the IPv6 address in the URL (ie 2a02:8071:2909:7b00:5604:a6ff:fe93:d1a6 )
It's too stupid to simply assume that it by default the not-special-case name is to be sent to the proxy
TimToady r: say "Hi, I'm running on $*VM<name>."; # don't need the braces 18:28
camelia rakudo-moar d78036: OUTPUTĀ«Hi, I'm running on moar.ā¤Ā»
..rakudo-jvm d78036: OUTPUTĀ«Hi, I'm running on jvm.ā¤Ā»
..rakudo-parrot d78036: OUTPUTĀ«Hi, I'm running on parrot.ā¤Ā»
nwc10 oh, sorry, that was supposed to be to #perl6 I think 18:29
TimToady ooops
nwc10 me, not you :-) 18:30
TimToady metoo 18:31
was responding to the #perl6 bah clog 18:32
dalek arVM: ac6a3fc | (Tobias Leich)++ | src/io/fileops.c:
do not always create target in copy

In case the soure does not exist, the target should be left untouched.		 19:21
FROGGS everybody should delete files like non-existent-copy-mtgt and non-existent-copy-stgt in his rakudo dir if he/she spectested moar in the past 19:25
dalek arVM: ba1af90 | (Tobias Leich)++ | src/strings/ops.c:
return empty string if pos > chrs in substr

Just to be in line with the other backends.		 19:38
diakopter FROGGS: have the clean stage do it?
FROGGS diakopter: what? I don't understand 19:40
diakopter make clean
or just add the deletions to configure :)
FROGGS hmmm, good point
diakopter [and remove them in a few months]
FROGGS diakopter++
19:40 jnap joined
tadzik oh, that might help panda 19:57
jnthn argh, readline works how... 20:46
diakopter a line at a time
jnthn a byte at a time AND THEN SEEKS BACKWARDS AND READS IT ALL AGAIN
No wonder it doesn't work on a tty...
FROGGS hides 20:49
jnthn grmbl
FROGGS also hides JimmyZ
nwc10 jnthn: this is our less than awesome code, or third party awesomeness?
was not wanting more names named than that
jnthn Ours 20:50
nwc10 oh. oops
diakopter ours, ish
nwc10 Oh my! 20:52
The original torture has made some progress
Stage start : 0.005
Stage parse : 1144911.468
diakopter O_O
nwc10 rn: say 1144911.468/86400 20:53
camelia rakudo-parrot 19ba6a, rakudo-jvm 19ba6a, rakudo-moar 19ba6a, niecza v24-109-g48a8de3: OUTPUTĀ«13.25129014ā¤Ā»
jnthn Wow
nwc10 OK. So it may well complete
give it another week or so
jnthn Stage parse survived the torture! :)
nwc10 Stage syntaxcheck: 0.068 20:54
Stage ast : 1.188
so now in optimize? I forget
FROGGS ohh wow
jnthn Yes, optimize now 20:56
Thing is, since you started, the time phase optimize takes has kinda halved :)
And the time stage mast does also has :)
FROGGS ctrl+c and try again?
>.<
nwc10 there are more cores
have already been doing other things in parallel 20:57
FROGGS jnthn: I shall take a look at readline then? Ć³.Ć² 20:58
20:59 jnap joined
jnthn FROGGS: Well, trying a few things wiht it here... 20:59
FROGGS k 21:00
jnthn C:\consulting\rakudo>type README | perl6-m -e "say $*IN.lines.elems" 22:00
121
phew
Though, what I've done is a total hack really. 22:01
Just that the larger re-work needed is mroe than I've time for.
FROGGS :/ 22:02
jnthn (as in, if I want $*IN not to be totally busted in the release) 22:03
timotimo why didn't anything happen while i was gone? :( 22:12
jnthn timotimo: It...did, I just didn't push it yet 22:13
timotimo ah, phew :) 22:16
dalek arVM: 55e543a | jnthn++ | src/io/fileops.c:
Do Just Enough so $*IN.get/$*IN.lines work.

This is *not* a good way to do, well, anything. However, pending the upcoming review of IO stuff after this release, it hopefully gets us a working $*IN.get and $*IN.lines. Does at least work in Windows. For now, regress on supporting a plain \r as line ending in readline; can put that back more easily after the future cleanup.		 22:17
jnthn FROGGS, timotimo: Please could one of you test that cat README | perl6-m -e 'say $*IN.lines.elems' works now after ^^
FROGGS sure
jnthn Only needs a Moar rebuild, not the whole stack 22:20
[Coke] Dirty trick, but moar pulled ahead today.
jnthn I think honest mistake more than dirty trick :) 22:21
[Coke] I prefer my version of reality!
FROGGS jnthn: it hangs 22:22
jnthn oh ffs
FROGGS let me take a look
jnthn FROGGS: Does say $*IN.get work? 22:23
FROGGS it says nothing
jnthn :/
I assume README exists in the directory you're doing this in? 22:24
FROGGS I am doing this with VERSION in moar 22:25
for README.markdown both callbacks get called for about ten times each 22:27
nread at line 300 is positive 22:29
the condition in line 302 is never met
there is something in read_char 22:33
ohh, it does not blok 22:35
block*
jnthn "it"? 22:36
dalek arVM: c57b446 | jnthn++ | 3rdparty/dyncall:
Update dyncall submodule for Solaris 11 patch.
 22:37
FROGGS well, it calls the callbacks after it already left the readline function
it == the program 22:40
jnthn wat 22:41
oh, I wonder if it's a refcount thing 22:42
FROGGS I see my printf() after uv_run() before the callbacks get called
jnthn grrr 22:43
FROGGS: Works on Windows... 22:46
FROGGS: Which is really odd
FROGGS: Got a patch for you to try
FROGGS sure
jnthn woulda expected failure on both...
22:47 btyler joined
jnthn gist.github.com/jnthn/71e1e9fd60a3be0f99e9 22:47
FROGGS: Maybe that helps...or maybe not 22:48
FROGGS jnthn++ 22:49
works!
btyler works on osx too :)
FROGGS almost... 22:50
.get works, but lines.elems still hangs
yeah, lines hangs at eof 22:51
btyler both of those worked ok here (OSX 10.9)
dalek arVM: f9d020c | jnthn++ | src/io/fileops.c:
Make readline a little less broke on non-Windows.
 22:52
FROGGS >.o
jnthn FROGGS: Any idea how/where it's hanging? 22:53
FROGGS yeah, it looks like if it hangs when there is nothing to read
it does not even call the a callback
so I guess we should switch to UV_RUN_NOWAIT, and check for zero length after that too 22:54
jnthn hang on... 22:58
23:02 Mouq joined
jnthn FROGGS: Please can you try gist.github.com/jnthn/baa8286627c4695be9e7 to see if it helps? 23:04
FROGGS jnthn: hangs too 23:05
.get still working
jnthn FROGGS: Really? Wow...
Do we ever see the UV_EOF? 23:06
FROGGS no
Mouq FROGGS: what's the sh line that hangs?
FROGGS we get stuck before line 340
MoarVM$ make install && cat README.markdown | perl6-m -e 'say $*IN.lines'
Mouq Ah :( 23:07
FROGGS looks like that on_read is not called when there is nothing to read
jnthn FROGGS: Could you printf the return value from uv_read_start? 23:08
FROGGS always zero
jnthn argh 23:09
FROGGS it reads the 42 bytes (last line of README.markdown), and then tries again, and hangs in UV_RUN_DEFAULT 23:10
jnthn Can you see what uv_is_readable on the handle gives?
FROGGS always 32 23:11
jnthn wtf 23:12
jnthn tries it with the NOWAIT 23:14
Yeah, that won't do it... 23:15
FROGGS reads github.com/mozilla/rust/issues/10237 23:16
jnthn Ah, nice find 23:18
Looks like libuv patched it...
yeah, and we don't have that commit 23:20
FROGGS hmm, I dunno how to update submodules 23:21
jnthn on it 23:22
oh... 23:25
FROGGS hmm?
jnthn Can you do git submodule status in your MoarVM repo?
FROGGS 083b4051dd65f82907e38c1bcea3cbe33b56dc9a 3rdparty/dyncall (remotes/origin/HEAD) 23:26
3e054c36294f9d8fc197c9d1b715ea2db334f6bf 3rdparty/libuv (v0.11.12-1-g3e054c3)
41bb5a7df157173e3555d2074ec9a61212fe85c0 3rdparty/linenoise (heads/master)
23:28 benabik joined
dalek arVM: ee5a5bd | jnthn++ | 3rdparty/libuv:
Update to latest libuv.
 23:30
FROGGS src/io/socketops.c:113:13: error: too few arguments to function ā€˜uv_tcp_bindā€™ 23:33
uv_tcp_bind(server, &bind_addr);
jnthn wonders why he didn't see that 23:34
FROGGS I just pass a zero as third arg (flags)
jnthn oh, I didn't make clean. d'oh
yeah, go for that
FROGGS jnthn: I made a realclean, because it still hanged 23:35
works!
jnthn As in, doesn't hang?
FROGGS MoarVM$ cat README.markdown | perl6-m -e 'say $*IN.lines.elems'
75
and I see the text
jnthn \o/
FROGGS .get works too
jnthn Bad news: when I bumped dyncall, it broke the ability to build it... 23:36
Gonna have to back that out :(
FROGGS jnthn: that is HEAD btw, so not applied the gist atm
jnthn which gist? 23:37
gist.github.com/jnthn/baa8286627c4695be9e7 ?
I think it's better design to have that gist.
FROGGS jnthn: okay, I apply and test 23:38
jnthn ok
I got the socket build fix locally too
Gonna do an nqp/rakudo build and spectest cycle with this lot
FROGGS okay, that gist works well too :o) 23:40
jnthn OK. One more thing. What happens now repl wise? 23:42
Mouq jnthn: Infinite '>' on nqp-m and perl6-m 23:43
btyler likewise, at head with that last gist applied
Mouq (on my end, anyway)
FROGGS jnthn: same, infinit ">\n" 23:44
jnthn OK 23:45
I'll try something for that in a moment 23:46
btyler er, sorry, needed a clean. with the 'baa828' gist I get an error during the build. gisting
jnthn I think that's the last thing I'd really like to fix pre-release
btyler gist.github.com/kanatohodets/c3e26...1-txt-L184 23:47
dalek arVM: e13e697 | jnthn++ | src/io/socketops.c:
Build fix for latest libuv.
arVM: 49f8eb8 | jnthn++ | src/ (2 files):
Further improvements to EOF handling.
 23:49
arVM: 872224b | jnthn++ | 3rdparty/dyncall:
Revert "Update dyncall submodule for Solaris 11 patch."

This reverts commit c57b446f1c69a99d299471f4dddedea61888c9ba. Sadly, there are configure issues after the update, even if the Solaris patch is wanted.		 23:52
jnthn btyler: Hopefully pulling that last bunch helps
btyler jnthn: yep! it did
jnthn FROGGS, btyler: Maybe one of you could try gist.github.com/jnthn/8570089 to see if it helps get a basically functioning REPL 23:57
btyler roger 23:58
yeah! works! 23:59
FROGGS > say(42);
42

Powered by Cro and the RakuĀ® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!