weekly Perl 6 status meetings with phase transitions: Wed 17:30 UTC | IR clogs at irclog.perlgeek.de/phasers/today
Set by moderator on 30 September 2011.
00:25 colomon joined 00:28 colomon_ joined 02:56 colomon joined 03:29 colomon joined 07:09 colomon joined 09:08 [Coke] joined 10:45 colomon joined 15:32 benabik joined 16:00 colomon joined 16:33 mberends joined
moritz what I did this week 16:51
* lots of ticket triaging, closing, testing, fixing: down to < 750 open bugs, also thanks to [Coke]++
* a bit of exception hacking: Exception now has a .backtrace emthod 16:52
* applied MAIN/USAGE patch by japhb++
* fixed tests that assume some dispatches will fail at run time, even though jnthn++'s smart optimizer can catch them at compile time
* created a version of the mandelbrot-color benchmark that uses native ints to increment some indexes; noticably faster 16:53
* implemented native num ops
What I'm doing right now:
* trying to clean up the mess I made by the test changes mentioned above 16:54
What I plan to do:
* more testing, more exception hacking, more assisting with the optimizer
* test my tests on niecza
EOR
[Coke] did: dug through RT queue, trying to triage the 1 year old+ tickets. down to <750 tickets, as moritz said, also < 700 if you discount testsneeded tests. fudging nom, niecza as time permits, keeping moritz honest. 17:01
EOR
jnthn DONE 17:04
Rakudo:
* Added 'is required' trait for parameters
* Better private method error reporting
* Implement .+"$foo" style dispatches
* Fixed lcm/gcd infinite recursion
* Added error for declaring placeholders in the mainline
* Implemented $$foo, @$foo and %$foo
* Improved performance of string ranges a bit
* Improved performance of list assignment a bit
* Tracked down and fixed various issues in list handling, especially MapIter, that caused performance issues. for 1..1000000 { $i++ } now runs ~40 times faster.
* Various other small fixes
Optimizer:
* Added line numbers to CHECK failures, and grouped instances of the same error
* Implemented basic inlining of multis and onlies in some simple cases; enough for many core ops, though not restricted to them
* Initial native operators; moritz++ added some more
* CHECK-time detection of various calls to only subs that could never possibly work
* Extended type analysis to catch more "could never work" situations at CHECK time
TUIT JAR
* My $dayjob was relatively quiet in September, but this and the next couple of weeks will be decidedly busier. After that, it'll quieten down again. I'll have tuits in the next couple of weeks, but less of them than I have in recent weeks. 17:05
WORRIES
nom currently has significant missing chunks in regex support, and some nasty list bugs. These are areas that pmichaud++ has tended to take care of, and I've not been so involved with; the division of focus has tended to work well. Unfortunately, Pm's tuits seem in short supply, I've little idea when that will change, and bugging Pm for patches is the last thing we should be doing.
While I can find and often fix a wide range of issues in Rakudo in O(minutes), for me to sort out list bugs is more on the scale of O(hours). Me being able to do significant hacking on the regex engine is more like O(days).
Mostly, your options if you want to see progress on list/regex issues are...
* Be patient and wait until Pm has tuits again
* Be patient while I try to piece together an understanding of these less familiar bits of Rakudo, so I can make things better. Yes, I understand lazy lists OK and I went to my finite automata class like a good computer science student, but these things are still a bit tricky to get in to. :-)
* Dive in and try to fix them ;-)
I'm somewhat "feeling the pressure" on Rakudo nom at the moment; things really aren't where I hoped they'd be at this point. I don't feel there's an immediate risk of that translating to burnout, though I may use my couple of few-day teaching trips in the next couple of weeks as short escapes from Perl 6 work also.
So, don't worry if I disappear for a couple of days, and thanks for understanding.
EOR
moritz++ [Coke]++ # amazing ticket work! 17:06
moritz jnthn++
jnthn Also moritz++ for company in optimizer land :) 17:07
[Coke] jnthn: you're doing the hard work, I feel like I'm just mucking out the stable. ;) 17:10
but you're welcome. 17:11
colomon Mucking out the stable, doesn't that make you Hercules? ;) 17:13
PerlJam reads jnthn's text above as "niecza ... now's your chance!" ;-)
moritz PerlJam: niecza suffers from the same underlying problem: too low bus number in many important areas
PerlJam yeah 17:14
As as no one goes the way of pugs ... 17:15
s/As/As long/
mberends The answer to low bus numbers is within each of us. When choosing how to use our precious tuits, let bus-number-lowering be a high priority. 17:16
moritz tries to keep the bus number on the test suite > 0.9
PerlJam mberends: part of the problem is that there is insufficient knowledge about the critical matters spread across our developers (IMHO anyway) 17:17
pmichaud my tuit supply is still... uncertain.
I may have some time on Friday; weekend looks unlikely; early next week is possible. 17:18
17:18 masak joined
pmichaud is there a list of missing regex features? 17:18
jnthn pmichaud: The one that scares me most is protoregexes.
pmichaud okay, other than that one?
masak pmichaud: I could play around a bit during the evening and see what I come up with. 17:19
jnthn pmichaud: backtracking, before and <&foo>
pmichaud I thought <&foo> was done.
masak pmichaud: there was a worrying backtracking bug the other day.
pmichaud: basically, quantifiers don't backtrack in captures.
PerlJam does the spectests no longer hang on infinite now? 17:20
jnthn pmichaud: oh, so it is.
pmichaud backtracking into subrules/subcaptures might not be implemented yet, no.
not hard for me to do, though.
jnthn pmichaud: oh...I got mixed up...I think it was actually passing args to regexes.
<foo(1,2,3)>
masak sounds familiar.
pmichaud did ng support defining regexes w/args? 17:21
or are we talking about passing args to methods and invoking them from regexes?
jnthn pmichaud: We depend on it in Perl6::Grammar
pmichaud: I mean, it supported it to some degree, as we used it Perl6::Grammar. 17:22
Also there's a spectest I'm pretty sure ng ran.
pmichaud oh, that's still using the old engine, though, yes?
jnthn Yes.
pmichaud okay
jnthn Also, there's some list issues.
pmichaud anyway, args to regexes/methods isn't a long coding effort (for me) 17:23
jnthn Yes, the "(for me)" comes up quite a bit in these things ;)
pmichaud assign me tickets and I'll hack away at 'em :)
masak pre-report: finally pushed my work so far on macros D1 to a branch in the rakudo repo. one known bug and a few possible improvements. will blog, but no idea when there might be time for that. no word yet on the grant application. EOR
jnthn pmichaud: for 1..$n { } was accidentally quadratic.
pmichaud "was"? 17:24
masak /o\\
jnthn pmichaud: Yeah. When my C profiler told me that for 1..1000000 { } was spending 95% of it's time in memmove, I went hunting. :)
pmichaud seems odd that it'd be quadratic, after I worked on making sure that for 1..20000 { } was reasonably fast.
moritz pmichaud: I just assigned you two tickets (one lists, one regexes) 17:25
pmichaud moritz++
jnthn pmichaud: In eager mode, we reify 100000 or so items at a time
pmichaud yeah, because that's faster than reifying 1 item at a time 100000 times :) 17:26
jnthn pmichaud: MapIter was doing .munch one of those at a time. Each time, that does a shift on an RPA
A shift on an RPA means memmoving everything in the RPA one element to the left.
A memmove of 99999 items takes a while. The 99998 takes quite a while too. ;-) 17:27
pmichaud so, what was the fix?
or is it not fixed yet?
jnthn pmichaud: Probably the wrong one :(
pmichaud: munch all we're going to work with in one go
pmichaud: And then just keep a position.
pmichaud hmmmm
TimToady
.oO(shifts are very cheap in Perl 5 by keeping an extra pointer...)
 17:28
Util Report: No Rakudo work this week; none expected next week due to $WORK.
pmichaud I'll have to look at that. the problem I see there is that means that MapIter might not be as lazy as it needs to be
Util Q: Date of next nom-based Rakudo Star?
EOR
pmichaud i.e., it'll consume too much
jnthn pmichaud: If that's very wrong, feel free to undo it.
pmichaud: Yeah, I worried a bit too
pmichaud: tbh, the main time was spent on understanding the list implementation more and diagnosing what was wrong.
Both of which are beneficial even if my patch doesn't hold up. 17:29
moritz so, it's #phasers time 17:30
pmichaud I'll have to think of a way to speed up MapIter there.
moritz any reports from people who haven't reported yet? 17:31
colomon sure, I've got one queued up here and ready to go
pmichaud report: tuit-starved here. Some tuits possibly on Friday. Weekend unlikely. Next week probable.
moritz colomon: fire then
pmichaud EOR
colomon Pre-report: (no longer pre- because I didn't want to interrupt the conversation)
* Mucked about with mberends++ GTK stuff in Niecza
* Started implementing my "tune reminder" project
* (Goal is to help remind me to practice the tunes I know)
* Currently it probably works with Nom, but I've been developing under Niecza...
* ...because I plan to use GTK for a user interface for it
* In the process, discovered open :w was not implemented and got a crude version of it working
Hope to:
* Blog on end result of trig work
* Blog on tune reminder
* Get tune reminder prototype up and reminding
EOR
PerlJam I have nothing to report other than I'm doing the compiler release this month. EOR 17:32
pmichaud PerlJam++
moritz colomon++
++PerlJam
pmichaud also tadzik++ and others++ for the release this past weekend
moritz so, questions
Util queued one, I also have one
mberends q1q
Util ETA for nom-based Rakudo Star? 17:33
moritz "I hope soon"
but currently many modules don't work on nom yet
pmichaud when there's a consensus that nom is ready for it
(which I don't think we have yet)
Util Fair enough.
PerlJam pmichaud: and then back to timed releases?
pmichaud PerlJam: yes.
PerlJam bueno 17:34
Util EOQ
moritz I hope we can make the compiler releases from now on timed and on schedule again
pmichaud probably monthly to begin with again, as we make rapid updates to nom and modules; then back down to 3-month again
moritz +1
pmichaud I think compiler releases can return to normal schedule now
moritz ok, my question...
how do we deal with exercising compile-time failures in roast? 17:35
problem is that eval_dies_ok evals in a different lexical scope
so symbols from the caller are not visible to it
so I went with nok eval('...'), 'description';
which breaks niecza, which correctly does not catch exceptions in eval() 17:36
so I tried nok try { eval '...' }, 'description';
and that breaks rakudo, because the return value from a failed eval() isn't a proper Perl 6 object
I get segfaults and stuff
pmichaud fix rakudo's eval()
moritz pmichaud: I did, in a branch
pmichaud: but it needs also try { } fixed 17:37
and i don't know how to do that
(branch eval-throws fwiw)
is a fix for try { } returning the exception (or a Failure wrapping the exception) possible in the nearish future? 17:38
(in case of an exception thrown, that is)
pmichaud that seems like it wouldn't be too difficult (more)
try should already be wrapping any exception it receives.
moritz the thing in $! is properly wrapped
pmichaud I suspect the problem is try's return value.
moritz correct
pmichaud that ought to be a simple-ish fix 17:39
tadzik 's here
moritz $ ./perl6 -e 'say try { die "foo" }'
Null PMC access in find_method('gist')
such a fix would be greatly appreciated
pmichaud right... I bet try isn't returning anything atm.
moritz EOQ
pmichaud thus the NPA
17:39 ashleydev joined
moritz mberends: you too had a question? 17:39
mberends my question: DateTime.now hangs on 32-bit nom, because of some data type problem. We have to comment out 5 test files in spectest.data as a result. This itch wants me to scratch it, but I'm worried that the problem may be pretty deep. 17:40
what to do?
moritz the problem *is* pretty deep
pmichaud I think out-of-range ints should autopromote to num
moritz it's that we don't have enough prevision in a Rat to do the Num -> Rat conversion on 32 bit, because our ints aren't big enough
pmichaud oh, but our Rats are using 'int', yes?
moritz don't think so 17:41
pmichaud oh
moritz nope, untyped objects
pmichaud but the constructors are typed.
hmm
moritz we *could* store Num's in there
pmichaud anyway, autopromote int->num is my suggestion. there's even an opcode to simplify it now.
moritz but we'd need to provide alternative, unsafe constructors
jnthn sorry, had phone call 17:43
pmichaud another completely bizarre possibility is to have our Int types box num natives
jnthn Well, real solution is bigints :)
pmichaud I don't see "real solution" in O(weeks) though.
if our Int types boxed num natives, we'd also have the ability to store Inf and NaN in Int
jnthn Not unless it becomes a priority for somebody, no.
pmichaud: Yeah but...they'd not really be Ints then ;) We could in theory try it but I dunno what the fallout would be :) 17:44
pmichaud I can't imagine that switching from int to num internally would pose that big a speed differential
colomon issue is precision
pmichaud why wouldn't they be "Int"s? Int is something of an opaque type anyway
jnthn I'm not so much worried about speed as I am about the int/Int relationship. 17:45
moritz well, we'd have to switch all those _i opcodes to _n
jnthn And it's precision as colomon mentions.
colomon a 64-bit int has more precision than a num can hold
pmichaud a 64-bit integer has more precision than an int can hold, on a 32-bit system :-P
jnthn One thought on this. 17:46
We're going to need a set of ops for handling bigints anyway.
I could potentially implement those just with 64-bit ints (even on 32-bit platforms) for now.
cotto q1q
jnthn And fill it in with a real bigint implementation later.
pmichaud jnthn: pondering 17:47
tadzik q2q
pmichaud what ops would we need? and do we still want to deal with 64-bit overflow somehow?
jnthn It's a kinda stepping stone to real bigint support. Which will have to come sooner or later.
tadzik (may overlap with cotto++'s)
cotto likely 17:48
jnthn pmichaud: The "usual set" :)
jnthn guesses tadzik and cotto have comments relevant to bigint :)
If so, go ahead.
cotto nope
jnthn oh :)
tadzik (:
pmichaud jnthn: ooc, how would you change the Int representation?
tadzik . o O ( q3q )
jnthn pmichaud: Int is a P6opaque. It's the representatioin it "inlines" that would need to change. 17:49
*representation
pmichaud jnthn: thus I'm asking, how would that changes?
currently we have
Int.HOW.add_attribute(Int, BOOTSTRAPATTR.new(:name<$!value>, :type(int), :box_target(1)));
jnthn That :type would almost certainly change. 17:50
pmichaud would it become :type(bigint) ?
and then we have some way of mapping "bigint" to (for now) 64-bit integer in C ? 17:51
sorear did: mostly hacked on /serialize. extreme tuit shortage now, I'll backlog the meeting later
jnthn pmichaud: Something like that. Though the name "bigint" would really just be a vessel for an appropriate representation.
PerlJam for jnthn's proposal, seems like it would be :type(slightly-bigger-int)
pmichaud we could always do int64
jnthn ooh :)
Yes, we could :)
mberends there is some almost ready bigint code in zavolaj/examples/biggishint.[ch] 17:52
jnthn mberends: ooh
pmichaud: Anyway, I'm thinking this really needs the same stuff I need to handle compact structs.
pmichaud jnthn: any idea of ETA? 17:53
jnthn pmichaud: Which needs me to work out the details of how that's going to look.
PerlJam jnthn: is that the same stuff needed for NCI (or FFI)?
pmichaud sounds like "not quick"
jnthn pmichaud: Depends what "quick" means :)
pmichaud: I suspect once I get a chance to think it through I'll work out how it should look quite quickly.
pmichaud jnthn: well, I agree there :) 17:54
jnthn pmichaud: It's an ideal task for a long train journey, and I've got four of those in the next 2 weeks. :)
pmichaud okay, so O(2 weeks) at least :)
jnthn So provided I have a functional brane on one of them... :)
Yeah. I'm not likely to get to much Perl 6 stuff in the next week or so :(
pmichaud okay -- if there's a ticket for the datetime 32-bit issue, assign it to me.
masak doesn't dare point out to the uni prof that that's not how you use O() 17:55
pmichaud if there's not a ticket, create one and assign it to me
masak submits rakudobug
pmichaud switching int to autopromote to num is fairly straightforward
moritz there is a ticket already
masak oh, ok.
standing down, then.
jnthn oh, q1q...the queue is getting quite long now :) 17:56
pmichaud I'll patch up Rat to accept integral Num values
with a big XXX beside it so we know we need to come back to it
jnthn pmichaud: Ah, just patching up Rat worries me a lot less than patching up Int.
pmichaud anyway, it'll get us around this issue (and others) on 32-bit systems for a short while yet 17:57
I'm done with this topic.
jnthn *nod*
moritz pmichaud: #100136 assigned to you
ok, cotto had a question
cotto tadzik mentioned that Rakudo was having some issues because an installed parrot couldn't be relocated. Can someone elaborate on that? 17:58
tadzik that's the reason Star cannot be packaged easily
masak seems to be mostly an issue for package managers.
tadzik it's also a reason why Rakudo on windows always has to be in C:\\Rakudo
cotto that makes sense now
[Coke] tadzik: I own a ticket about that. I disbelieve. 17:59
(has to be C:\\ on windows)
I agree you probably cannot build once, move anywhere, though.
jnthn A Windows user
He wants his Rakudo here
But Parrot says no
:)
masak jnthn++
pmichaud that's a long-standing Parrot issue -- parrot installs aren't relocatable 18:00
cotto It's worth looking at how to fix in Parrot.
jnthn
.oO( if in doubt, explain with a haiku... )
[Coke] A terse bug report
tadzik Haikus are easy. But sometimes they don't make sense. Refrigerator
[Coke] with very little detail
moritz jnthn: it misses the reference to a time of your to be a proper Haiku :-)
[Coke] unlikely get fixed
TimToady Refrigerator obviously refers to the time of midnight snacks 18:01
masak tadzik: it's the only autopun haiku I know.
jnthn I dunno about other platforms, but Windows users tend to expect they can download a binary installer, and tell it where to install ot.
*to
moritz well, linux users like that too :-) 18:02
jnthn OK :)
masak Mac users don't care.
jnthn masak: A general statement? ;-)
masak or rather, they like everything in /Applications/
Util If Parrot fixes its relocation problems, will that completely resolve Rakudo's reloc issue?
tadzik I suppose so 18:03
moritz Util: it might not immediately, but it's a required step
jnthn Util: We install stuff in the "standard" Parrot places afaik.
So there's a decent chance it'll get us much of the way there.
TimToady absolute paths can sneak in at any level
tadzik the problem with Star is "we can't build modules before we install Rakudo"
moritz cotto: question answered? 18:04
cotto moritz: yup
thanks
moritz then on to tadzik's questions
tadzik first one is already answered
I'll now stick in my short report:
TimToady wow, that's, like, 0 lines :) 18:05
cotto actually, would an environment variable be enough to make windows users happy, a la PERL5LIB
tadzik done: .mobi version of Using Perl 6 book, required a few fixes in the result of 'make html', not sure if it's our fault, or PseudoPod's. Got some feedback, will probably give it more love
also, some minor patches and some ticket mangling, nothing worth mentioning I suppose 18:06
now, to the question: release went flawlessly, but I received some negative feedback from people packaging Rakudo for Linux distros. The problem is that Rakudo 'Riga' NQP_REVISION was newer than NQP 2011.09 release 18:07
we give much care about releasing Rakudo, but I suppose we have to give at least some of that love to NQP releases as well
TimToady cotto: env vars can present security issues
moritz tadzik: agreed
tadzik I can find the relevant irclog
pmichaud should've had a 2011.09.1 release, I suspect. 18:08
cotto I'll see what we can come up with in Parrot and check back when we have something for Rakudo to play with.
tadzik afaik, but not sure, what stopped the maintainer from just using --gen-all-the-things was the relocatibility issues
moritz ... or the nqp tarball I uploaded to perl6/nqp/downloads
... after the issue
PerlJam tadzik: are linux dists packaging nqp separately? 18:09
tadzik PerlJam: it appears so, yes
pmichaud rakudo (nom) releases should always target a nom release
if that means we have to issue a separate nom release, then do that :-) 18:10
tadzik not sure what was the blocker for using just --gen-nqp with previously installed Parrot
moritz tadzik: distro policy
tadzik pmichaud: s/nom/nqp/?
oh, right
makes sense
pmichaud s:2nd/nom/nqp
I'll rewrite.
nom releases should always target a nqp release. If that means we have to issue a separate nqp release, then do that. 18:11
tadzik makes sense
pmichaud I suspect the release guide needs to be updated with that detail.
PerlJam Um ... when do NQP releases happen?
moritz pmichaud: aye
pmichaud PerlJam: between the parrot and rakudo release :) 18:12
PerlJam (oddly, I'd never considered "nqp releases" until this conversation)
pmichaud we've mused about nqp releases before. we've also thought about extending the rakudo release to be "up to 4 days after parrot release" to give time for nqp stuff to happen.
PerlJam how about just make the rakudo release the following week? 18:14
pmichaud anyway, I'd say that nqp releases happen just prior to or in conjunction with rakudo releases
we can separate the timing if we decide we need to do that
(separate the timing later)
I'm fine with saying "rakudo release following week", too. given that parrot has talked about making changes in their release/deprecation cycles, it makes sense for us to rethink the rakudo/star/nom cycles as well 18:15
s/nom/nqp/ # grrrr 18:16
tadzik: I know it's hindsight, but would a separate 2011.09.1 nqp release have helped the situation?
tadzik pmichaud: yep
pmichaud then that should be our policy going forward, I think. 18:17
tadzik pmichaud: moritz++ uploaded another tarball and it fixed the issues
that should be the Justin Case policy for sure
but I'd rather have no problems than having to fix problems
pmichaud well, I think we have to recognize that from time to time we'll want to make .1 and .2 releases
things happen.
tadzik sure 18:18
sorear moritz: we changed the spec there, eval is no longer supposed to catch exceptions 18:19
pmichaud okay, I'll make sure to check that the release guide gets updated. (If someone wants to draft updates for me to check, that'd be awesome)
moritz sorear: I know
pmichaud (thus I said "fix eval()" :-)
moritz sorear: I'm sorry for having busted so many spectests for niecza
sorear: I'll fix it, but it might need some time 18:20
pmichaud forgiveness > permission, I think :)
sorear and I'm now backlogged on #phasers.
if anyone has a question for me to answer in the next 5 minutes...
pmichaud same here -- I will have to depart shortly 18:21
moritz I think jnthn++ had one more question
jnthn Yes
Optimizer branch. I want to merge it at some point BUT would prefer to avoid chaos. :)
I'm pondering something like this. 18:22
There's a --optimize option. It defaults to "1". We start of classifying any non-very-comfy optimization as "2". We build the setting with --optimize=2 because we have good test coverage of the setting.
That way we immediately get the CHECK time analysis. 18:23
masak sorear: how far along are you with the /serialization branch?
pmichaud jnthn: I have no objection to that.
masak sorear: and what will it mean, exactly, when it lands?
moritz jnthn: +1
jnthn Over time, optimizations get migrated into "normally on" as we're comfortable with them. --optimize=2 always gives you the "most benefits, most risk"
And --optimize=0 totally disables it, so we can know if the optimizer is guilty of causing issues :) 18:24
pmichaud I might suggest --optimize=1|2|3
3 is the "most benefits, most risk"
2 is the default, the ones we're comfortable with
1 is "very stable optimizations" -- i.e., those that are known to be good based on experience 18:25
jnthn pmichaud: That also works for me.
pmichaud i.e., in case we make a mistake moving one from 3 to 2, the end-user still has an option for --optimize=1
jnthn *nod*
benabik And -Ofun gives which optimizations? ;-)
pmichaud (without losing all optimizations)
jnthn pmichaud: Good point. Thanks, will do that.
I need to clear up a few things, but it's potentially landable within a week. 18:26
pmichaud +1
jnthn It does optimize the setting in various places. 18:27
Note that we have an incentive to (carefully) add a bit more type info in the setting after this lands.
Just need to be careful not to over-constrain things.
pmichaud or to supply suitable multi candidates :) 18:28
moritz now that we have ops for native ints in the setting, it might be a big win to change some of the loop variables to int
jnthn If there's $a + $b and it knows both are Int, for example, then it will inline the addition, and not have to make a dispatch at all.
moritz: Yes, it is
tadzik++ posted some numbers on this not so long back.
It's an order of magnitude faster.
moritz it shove several seconds off of the mandelbrot benchmark 18:29
jnthn Yeah, the MapIter change also did...wonder what happens if we combine that :)
*those
moritz and since ops of natives are often inlined, we can make the Complex stuff less ugly
jnthn: that loop didn't use MapIter at all
jnthn moritz: mandelbrot uses MapIter a lot though. 18:30
moritz: I meant, that change plus the optimizer together might bring us down under 40s or something :)
moritz I think we have reached EOP (where P = Phasers) 18:33
tadzik may be
jnthn pmichaud: oh, just found these: gist.github.com/1265262
pmichaud: Numbers before/after my MapIter change.
pmichaud jnthn: okay, I'll definitely think about how to fix up MapIter then 18:34
we definitely need it to be able to preserve laziness, though.
jnthn pmichaud: My reasoning was "if we're already eager..." - but I can see how that may not be enough. 18:35
pmichaud ...how do we know that we're eager, though?
and how do we know that the body doesn't abort in the middle?
(thus losing the munched-but-not-processed items from the list)
jnthn pmichaud: I took care to make sure we don't lose them in the patch. 18:36
pmichaud: But we do certainly over-munch in that case.
pmichaud the point being that they're no longer on the list, I think.
(source list)
that might actually be okay; I have to think a bit about the .map setup. 18:37
jnthn pmichaud: I stick 'em back into the source list.
pmichaud even with an exception?
hmmmm
jnthn Look for "uneaten_saved" in the patch :)
:)
masak .oO( zombies ) 18:38
jnthn pmichaud: I'm not attached to the patch. I am kinda attached to the speed improvement though ;)
pmichaud I agree we need the speed improvement.
but correct is still as important as speed :)
jnthn I agree we need the right semantics. :)
I mostly hoped I could get away with the things my patch did, and the spectest didn't tell me I couldn't. :) 18:39
pmichaud well, it's one of those things I suspect is underspecced
it may very well be that MapIter ends up needing a very close relationship with List
jnthn If it's allowed to go peeking, it may have an easier time being efficient, yes.
pmichaud for the version I put together, I tried to keep a little distance in case we're MapIter'ing something that isn't a List. 18:40
jnthn *nod*
Oh, I also noticed that we seem to end up wrapping things in two levels of List/Parcel when doing a for loop
pmichaud often that's true, because we need to .flat
so one of the List's is the part that does flattening 18:41
jnthn OK, I was probably missing something; it looked like one level too many.
But I'm still figgering it all out :)
pmichaud It may be one level too many, yes.
but we have to ensure that things are flattened properly (more)
jnthn I quite like it as I'm working through it; it's just a little mind-bending at times ;) 18:42
pmichaud also, believe it or not, MapIter comes from before when .map was specced to always .flat
jnthn oh :)
pmichaud now that .map always flattens, we may be able to optimize a bit further than I was dealing with at first.
(instead of wrapping in a flattening list)
it's very possible that .map simply becomes a method on List that is really smart about iterating. 18:43
(and that MapIter may go away entirely)
masak \\o/
pmichaud i.e., perhaps ListIter can be given the mapping smarts. 18:44
jnthn Well, if MapIter were going to have such a cosy relationship with List anyway, that could make a lot of sense.
pmichaud right
otoh, MapIter tends to want to consume its list, while ListIter doesn't
jnthn yeah, it munches.
pmichaud I'll think about it a bit more 18:45
jnthn OK :)
pmichaud I agree the O(n^2) memmoves is a problem that we have to avoid.
(did I get that O(...) correct this time, masak++? ;-)
jnthn Yes. The C profile was...incredible.
pmichaud we really do want to be shifting things somewhere 18:46
otherwise we end up with huge map memory footprints
i.e., for 1..1000000 { } we'd really prefer not to keep the 1-million element list around for the whole loop, or to construct it all in the first place 18:47
jnthn Yes, but we could throw build and throw it away in 10000 element increments.
pmichaud right
jnthn s:1st/throw//
pmichaud that's what the old implementation did -- it tended to limit itself to smaller increments
jnthn I guess "how big" is a matter of tuning. 18:49
Or we find a way to shift cheaply :)
pmichaud or munch cheaply
jnthn Which may mean moving away from RPAs.
pmichaud we can keep RPAs if munch becomes cheap.
jnthn Or keeping an RPA and an index.
pmichaud I think making munch cheap is the bigger win. 18:50
(which likely involves an RPA and an index, yes.)
jnthn Yeah, munch is where we were being killed.
well, our performance was :) 18:51
pmichaud then we can leave MapIter alone-ish.
jnthn That said, github.com/rakudo/rakudo/commit/3b...779a8cc2bb also helped a lot :)
pmichaud heh
jnthn Actually I still need to go back and fix that..all the pushing leads to a load of time in realloc. 18:52
pmichaud I think shiftpush was written before I "fixed" RPA's splice to not be quadratic
jnthn RPA's splice used to be...omg!
pmichaud RPA's splice was quadratic when I started working on lists...
jnthn o.O
pmichaud just a sec, I'll find the patch :)
jnthn Please say splice wasn't implemented in terms of shift and push... :)
pmichaud github.com/parrot/parrot/commit/f1...6f31683aa9 18:53
it was implemented in terms of keyed_int
jnthn Still, youch! 18:54
pmichaud yeah
so if you were inserting at the beginning of a list, you ended up doing O(n) vtable calls via keyed_int.
instead of a memmove
jnthn ugh
pmichaud so shiftpush was written because splice was not yet fixed in whatever version of Parrot I was working with :) 18:55
(also because it's quite convenient)
jnthn It is also quite convenient :)
:P
pmichaud okay, I'll work on fixing munch. 18:56
jnthn OK, great. :)
pmichaud I suspect that if we simply keep a pointer, and the splice whenever the pointer reaches a certain point or we're doing more reification, that'd be enough.
s/the/then/
jnthn Sounds reasonable. 18:57
Oh, one of the list bugs seems to happen when you end up iterating a list that's currently being reified.
$!rest then ends up as Mu rather than an RPA
pmichaud yeah, there's an incorrect initialization or reset there somewhere
assign me a ticket :) 18:58
jnthn Yeah, I looked a little yesterday, but to no avail
OK
pmichaud okay, gotta run here
be back... sometime
jnthn ok, take care o/ 18:59
pmichaud (probably fri morning)
moritz \\o
19:18 masak left 21:19 benabik left 22:29 [Coke] joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!