Parrot 2.7.0 "Australian King" Released! | parrot.org Log: irclog.perlgeek.de/parrot/today | Nopaste: nopaste.snit.ch:8001 | close 25 tickets; remove deprecated items (especially CodeString); record deprecations; polish for 2.8 release
Set by moderator on 14 September 2010.
cotto takes $3.0 from whiteknight 00:00
whiteknight done
bacek_at_work cotto, did you miss "M" after $3.0?
chromatic msg atrodo What do we need to do to improve JONSLORITO ?
purl Message for atrodo stored.
aloha OK. I'll deliver the message.
whiteknight is fast. Like a ninja
cotto I'll start small. 00:01
dalek rrot: r49005 | luben++ | trunk/src/call/args.c:
Use plain hashes, not Hash PMC for internal named args processing
 00:03
rrot: r49006 | whiteknight++ | trunk/docs/project/release_manager_guide.pod:
+cotto
whiteknight cotto: What's the status of the gsoc_instrument branch? 00:05
I can say that it fails in parallel make. Regular make works fine 00:06
NotFound There are a bunch of throws in that function, and the flow is not enough clear to know if some is never reached with named_used_list. The safe side is to check all until lne 1107 00:08
chromatic Turn on the Coke-signal and let him check the dependencies.
whiteknight na-na-na-na-na-na-na-na Coke MAN! 00:09
Coke MAN!
feel lucky you can't hear the singing 00:10
chromatic I already GCd it. 00:11
whiteknight :) 00:12
NotFound If some day someone is able to rewrite that function in a cleaner and shorter way, I'll proclam he or she King or Queen of the parrots.
cotto whiteknight, safe to nuke 00:13
chromatic Is there a cash reward?
whiteknight cotto: nuke? I want to love it and merge it
NotFound: which function? 00:14
NotFound chromatic: you can put taxes on your subdits.
whiteknight oh, named_used_list
luben Done, I am freeing the strunct on all exceptions
NotFound whiteknight: fill_params
cotto The code lives on github atm, though I don't think khairul's done much with it.
It's definitely an awesome piece of code.
I love having a PMC that's a runloop. 00:15
s/loop/core/
whiteknight Ah, okay. I misunderstood you. So the branch can go, but the code lives on github. Gotcha
cotto yup
chromatic Why is the code on Github and not merged?
cotto github.com/khairulsyamil/parrot-instrument
It's more like an external project than a core feature. 00:16
Do you think it'd be better in core?
chromatic It'll break less in core.
Also get used more.
davidfetter +1 for core, fwiw
cotto this is true
seen khairul 00:17
purl khairul was last seen on #parrot 22 days, 19 hours, 55 minutes and 37 seconds ago, saying: hio cotto. i'm slowly moving the code over to here, github.com/khairulsyamil/parrot-instrument [Aug 23 04:21:36 2010]
aloha Sorry, I haven't seen khairul.
whiteknight I thought the code was pretty deeply integrated into libparrot 00:18
maybe thts a datd understanding of it
cotto yr mssng sm vwls 00:19
dalek rrot: r49007 | luben++ | trunk/src/call/args.c:
free internal hash struct on exceptions
 00:20
NotFound Another fine project: add sms encoding to parrot strings
Extra point for dcoding.
davidfetter /m lrnd hrbrw t n rly g, hnc hs n sr tm rdng thngs wtht vwls
whiteknight yeah, I think my keyboard is going all 'tarded 00:21
ok. I see what this project does. Different from what I thought 00:22
still cool though
nopaste "luben" at 192.168.1.3 pasted "Hash sizes stats" (13 lines) at nopaste.snit.ch/23343 00:23
whiteknight 40k hashes of size 0? FAIL 00:25
cotto That's unexpected. 00:27
00:28 patspam left
luben yes, 00:29
whiteknight luben: can we figure out where those wasted hashes are coming from? 00:31
luben they come as Hash PMC
most probably
whiteknight can we make Hash PMC more lazy? 00:32
chromatic 80% of hashes have 0-2 keys?
plobsing luben: what are those stats for? perl6 startup?
luben I hava made hash struct allocation lazy 00:33
plobsing, yes
whiteknight luben: in trunk?
luben yes
whiteknight so you've already made it lazy, and we're still seeing these numbers?
luben index/buckets area is allocated in lazy manner
hash struct is not allocated lazy 00:34
whiteknight ok
luben may be I could look in Hash PMC and see if it could allocate hash struct more lazy 00:35
chromatic Easily.
NotFound Maybe will be more productive to know what parts are creating lots of unused hashes. 00:36
chromatic Also true. 00:40
luben T think that's rakudo that creates them
chromatic Empty namespaces?
plobsing empty lexpads?
chromatic That could do it.
NotFound Sound likely
chromatic Sub's invoke() seems to create them only when necessary though. 00:41
luben args processing also creates hashes only when needed 00:42
chromatic Running cachegrind to see. 00:43
plobsing cachegrind can get you that info?
chromatic It can show the call paths that create hashes. 00:44
plobsing I suppose when 80% of hashes are near-empty, the largest allocator of hashes is very likely a large allocator of near-empty hashes.
chromatic Lots of LexInfos thawed. 00:45
luben chromatic, I am exploring callpaths with kcachegring also
00:47 cognominal left
chromatic Could be the thaw of Subs. 00:47
plobsing yes, but why do subs have lexinfos frozen in? seems like that's a problem in IMCC.
chromatic I'm not sure they do, but I don't understand the freeze/thaw in Sub very well. 00:48
plobsing VISIT_PMC_ATTR(INTERP, info, SELF, Sub, lex_info);
if lex_info is null, it will be stored and thawed as null
00:49 cognominal joined
chromatic Ah. 00:52
CallContext's exists_keyed*() VTABLEs.
They shouldn't call get_hash().
... at least, they shouldn't autovivify the hash.
NotFound There is a ticket about deprecation of autovivifys. Time to ge rid of them? 00:54
They can probably create a lot of empty unused hashes 00:55
00:56 davidfetter left
NotFound The probability of breaking things is not negiglible, though. 00:56
Big problem: named_used_list is freed in line 1139 but is used in 1182 00:59
luben NotFound, looking 01:00
pmichaud empty hashes: my guess would be named slurpy parameters where no named arguments were passed. 01:03
luben NotFound, shipped a fix 01:09
NotFound luben: looks good 01:10
dalek rrot: r49008 | whiteknight++ | branches/hash_allocator:
luben++ committed some things to trunk that this branch was working on. Useless now
rrot: r49009 | luben++ | trunk/src/call/args.c:
fix using a hash after it is freed
luben thanks for catching the bug
cotto How comfortable is tcurtis with git? It looks like he's on the schedule for the first release manager after the git move.
whiteknight cotto: trial by fire? There will be plenty of support personel 01:12
cotto If he's not solid, I'd try to sucker dukeleto for it.
01:13 kid51 left
whiteknight do you really think some ninja-level git-fu is going to be required for a release? 01:13
Most of the release process is filling out metadata and running fulltest
cotto not in general, just for the first one after the move
in case we missed something 01:14
whiteknight If it involves more than making and pushing a tag, I'll be surprised
pmichaud fwiw, one huge advantage of git is that anyone can try out the release process in full.
(at any time)
whiteknight true
cotto excellent point
01:14 Patterner left
pmichaud we found that out with rakudo; essentially, anyone can clone the rakudo repo and walk through the entire release process to see if there are any problems. 01:14
without having to affect the canonical repo 01:15
cotto dukeleto (or anyone who cares) can do a dry run before 2.10 comes out to make sure there aren't any omissions
pmichaud I would highly recommend that.
cotto git++
whiteknight maybe we'll all drop the ball in a major way, and we won't be on git for 2.10
pmichaud also might be worth trying it out with the 2.9 release
i.e., clone a git repo of parrot and pretend like one was doing the 2.9 release using git. update the docs from there. 01:16
cotto we might even consider moving to got before 2.9 if all the pieces are in place 01:17
whiteknight or, if we do it in the present-tense, we can move to git by then
:) 01:18
cotto either way, we're not moving until all the pieces are in place
01:23 rurban joined 01:32 esskar left 01:33 esskar joined
plobsing it appears that anything with an :outer gets a lexinfo 01:37
whiteknight yeah, I suspected something like that would be the case 01:38
NQP is pretty good about not creating :outer subs if lexinfo isn't needed
cotto seen mj41
purl mj41 was last seen on #parrot 49 days, 8 hours, 8 minutes and 41 seconds ago, saying: Lost connection to server irc.perl.org. [Jul 27 17:30:16 2010]
aloha mj41 was last seen in #perl6 1 days 9 hours ago joining the channel.
luben I have made callcontext to not create hashes when not needed (in get_X_keyed_str vtables) 01:40
plobsing whiteknight: !metaclass_method_forwarder gets an outer but has no .lex vars. why does it need a lexinfo?
luben empty hashes are down to 28k
whiteknight plobsing: I'm not familiar with that function 01:41
luben now I'll test it and ship
whiteknight blarg. parrot-instrument does't build against trunk 01:42
cotto khairul was working on using distutils when I last checked. I think the effort was incomplete. 01:43
whiteknight plobsing: ping 01:44
cotto: haven't heard from him in a while?
cotto nope 01:45
whiteknight that's disappointing
cotto he's probably busy with school
plobsing whiteknight: pong
whiteknight plobsing: parrot-instrument is having some build issues involving the interp struct. Do you know what happened with interp->save_func_table?
cotto He didn't give me a commit bit, but forking is easy 01:46
01:46 s1n left
whiteknight cotto: I already forked it 01:46
cotto commit bit plz?
plobsing whiteknight: interp->save_func_table got moved to code_segment->save_func_table
after dynops mapping, each cs has its own func table (to accomodate different dynop load orderings)
whiteknight cotto: done 01:47
luben When I should use GET_ATTR_attrname and when GETATTR_PmcName_attrname ? 01:48
whiteknight plobsing: can you take a quick look at this function? github.com/Whiteknight/parrot-instr...e.pmc#L704
luben: the former is for private access inside the PMC itself
the later is for general access everywhere else
luben OK, thanks
plobsing GET_ATTR_attrname could be ambiguous outside the .pmc file so isn't accessible
cotto the short form is mangled into the long form by pmc2c for VTABLE functions 01:49
and I think for METHODS too
01:50 hercynium left
plobsing whiteknight: detecting whether a dynoplib has been added has become easier - interp.n_libs will have incremented 01:52
whiteknight plobsing: okay, I figured things had changed there. I haven't been keeping up with your oplibs work like I should have
I think I need to fix the setup.pir code here first to make this crap build, then I can go about fixing/debugging the C
plobsing whiteknight: bus number is dangerously low there.
more eyes welcome 01:53
whiteknight plobsing: I'll do my best to familiarize myself with it. Do you have any docs or summaries to point at, or just the POD in the files?
dalek tracwiki: v31 | cotto++ | GitMigration 01:54
tracwiki: give git migration more solid dates, add tentatively responsible parties
tracwiki: trac.parrot.org/parrot/wiki/GitMigr...ction=diff
plobsing whiteknight: there's not a lot of doc. I've added functions and put a little POD in where appropriate, but there really isn't anywhere obvious to put documentation for this
whiteknight okay, that's fine. I'll meander around and see what I see 01:55
but it's late now, so I'm heading to bed. Laterz
01:56 whiteknight left 01:58 s1n joined
dukeleto hola 01:59
purl niihau, dukeleto.
dukeleto gives purl a penny 02:00
dalek rrot: r49010 | luben++ | trunk/src/pmc/callcontext.pmc:
Do not create empty hashes in get_X_keyed_str vtables
 02:01
cotto dukeleto, how does the git migration schedule look as far as stuff I put you down for? 02:02
nopaste "luben" at 192.168.1.3 pasted "updated hash usage stats" (13 lines) at nopaste.snit.ch/23346 02:05
cotto thinks of a better approach 02:09
chromatic parrot_hash_clone(INTERP, hash, get_hash(INTERP, dest)) ? 02:10
ah, that's fine
luben just the formating is not ok...
I missed a tab 02:11
chromatic I was thinking something else, but I was wrong. 02:13
Hm, 3.18% faster Rakudo startup in the past few days. Very nice. 02:15
plobsing luben: how are you instrumenting the hashes? 02:19
luben just printing to stderr in hash_destroy and counting 02:20
02:24 chromatic left 02:25 cognominal left 02:35 janus left, chromatic joined, janus joined
chromatic luben, I only see 33,053 hashes created for Rakudo startup. 02:42
... but 94,201 destroyed.
atrodo chromatic> well, I'm working on a default lookup method. after that, it should be ready to start playing with writing other pmcs 02:43
and how does it do that?
cotto that's very ... destructive 02:44
chromatic Sounds good. 02:46
luben In my callgrind profile parrot_create_hash was called 93486 times. hash_destroy 93152 times
chromatic Strange. 02:47
purl But true.
cotto Isn't there a way to instrument functions so that valgrind will treat them as a malloc/free pair and track any that leak? 02:49
or would that be redundant, since hashes have a chunk of memory associated with them 02:50
plobsing cotto: isn't that what PARROT_MALLOC is supposed to do somehow? 02:52
if a sub doesn't have any .lex declarations, does it need a lexinfo (it would be empty in this case)? 02:58
03:00 davidfetter joined
chromatic It shouldn't, no. 03:00
plobsing luben: can you recount empty hash creation after r49011 (requires rakudo rebuild to take effect)? 03:01
luben ok 03:02
03:03 Patterner joined
luben plobsing, 25294 03:07
plobsing huh, not as much as I had hoped
dalek rrot: r49011 | plobsing++ | trunk/compilers/imcc/pbc.c:
avoid creating unnecessary lexinfos for subs with :outer s
 03:08
nopaste "luben" at 192.168.1.3 pasted "updated hash stats" (13 lines) at nopaste.snit.ch/23348 03:11
dalek tracwiki: v32 | cotto++ | GitMigration 03:19
tracwiki: trac.parrot.org/parrot/wiki/GitMigr...ction=diff
chromatic Still an improvement. 03:30
purl an improvement is external to the program contruct.. knuth may be saying there is no advantage going multithread inside the program?
plobsing what made all the other hash stats drop? 03:33
other being non-empty
chromatic Smartening up CallContext.
03:35 davidfetter left
luben how do I regenerate core_ops.c ? 03:41
plobsing ./ops2c --core
cotto nice response time
luben yes, plobsing++ 03:42
plobsing muscle memory
purl i heard muscle memory was sometimes cruel
luben now, how do I regenerate vtable.h ? 03:57
plobsing that one I've never had to do
luben I am ripping deprecated logical operation vtables 03:58
plobsing woohoo! good ridance!
chromatic Modifying vtable.tbl should suffice. 04:00
luben I think I found how :) tool/build/vtable_h.pl
oo, yes, it is automatically rebuild 04:01
04:06 nwellnhof_ joined 04:08 hercynium joined 04:09 nwellnhof left, nwellnhof_ is now known as nwellnhof 04:40 hercynium left 04:49 tcurtis joined 05:13 rurban_ joined 05:15 rurban left, rurban_ is now known as rurban 05:30 theory left
dalek tracwiki: v2 | cotto++ | GitHubTracPluginTests 05:35
tracwiki: reorganize a bit, add another test case
tracwiki: trac.parrot.org/parrot/wiki/GitHubT...ction=diff
rrot: r49012 | luben++ | trunk (10 files):
Ripped out deprecated logical VTABLES
 05:40
rrot: r49013 | luben++ | trunk/src/pmc/callcontext.pmc:
fix indentation
luben When removing deprecated struff, does it needs to be removed from DEPRECATED.pod? 05:41
chromatic yes 05:46
plobsing luben: yes. also put it on the wiki.
follow the steps at "How To Deprecate" 05:47
luben plobsing++
cotto Ideally there'll already be a notification on the wiki. Don't assume it though. 05:48
luben++ for ripping out some cruft
luben I am reading the thicket again. As I understand it, it suggest also removing logical ops pn PMC
cotto did you check that it doesn't make Rakudo cry?
luben cotto, yes 05:49
cotto good for you
luben I have converted logical ops with PMC args to continue to work, they use VTABLE_get_bool internally
should I remove them? 05:53
cotto is glad his dayjob mostly didn't involve manual testing 05:58
dukeleto cotto: yeah 06:04
we really need to remove the SVK info from parrot.org 06:08
updating parrot.org/download is another thing for the migration 06:09
cotto I'd probably be using svk if it'd had a release in the recent past.
+1
purl 1
cotto botkick
dalek tracwiki: v3 | cotto++ | GitHubTracPluginTests 06:10
tracwiki: flesh out most of the tests
cotto dukeleto, are you adding it?
dalek tracwiki: trac.parrot.org/parrot/wiki/GitHubT...ction=diff
dukeleto cotto: just did 06:12
cotto axesome
dukeleto cotto: nice work on the github trac plugin tests 06:13
cotto thanks
dukeleto worked with OSU+particle to get smolder working, but it is still broken 06:16
cotto dukeleto, who'd be the best person to take point for the various bits of the git migration that'll need osuosl coordination? 06:17
dukeleto cotto: not sure. hanging out in #osuosl on freenode helps 06:18
cotto joins
06:21 nwellnhof left
dukeleto i just gave one of the OSUOSL guys the info for the user we need created 06:26
we will help him fix it tomorrow, hopefully
dalek tracwiki: v33 | dukeleto++ | GitMigration 06:27
tracwiki: trac.parrot.org/parrot/wiki/GitMigr...ction=diff
rrot: r49014 | luben++ | trunk (2 files):
Updated DEPRECATED.pod, bumped PBC_COMPAT
 06:31
06:33 uniejo joined 06:42 bacek_at_work left 06:45 bacek_at_work joined 06:46 bacek_at_work left 06:51 bacek_at_work joined 07:02 jsut_ joined 07:07 jsut left 07:14 chromatic left 07:31 luben_work joined 07:32 luben_work left, luben_work joined 07:33 tadzik joined
dalek kudo: 3f6392d | patrickas++ | src/core/operators.pm:
Avoid copying all of the lhs since we only need elements off of it
 08:05
bacek aloha, humans 08:25
08:36 tcurtis left
dalek rrot: r49015 | bacek++ | branches/gc_massacre/src/gc/gc_ms2.c:
Rework triggering of GC MS2 by rough counting of allocated memory since last collect
 09:54
sorear bacek++ picking up the GC 09:55
bacek ooookey... 10:32
We do need properly encapsulate "String GC". Without "shared buffer" Rakudo consumes too much memory for small substrings.
msg whiteknight We do need properly encapsulate "String GC". Without "shared buffer" Rakudo consumes too much memory for small substrings. 10:33
purl Message for whiteknight stored.
aloha OK. I'll deliver the message.
dalek rrot: r49016 | bacek++ | branches/gc_massacre/src/gc/gc_ms2.c:
Count PMC Attributes in used memory.
 10:44
wow hello 10:51
11:20 contingencyplan left 11:45 s1n left 12:05 tadzik left 12:33 bluescreen joined 12:37 patspam joined 12:57 sjn joined
sjn hey folks 12:58
can anyone here tell me if Allison is on IRC? (and if so, what's her nick?)
!seen allison 12:59
seen allison
purl allison was last seen on #parrot 16 hours, 32 minutes and 39 seconds ago, saying: dukeleto: great (on porting)
aloha allison was last seen in #parrot 4 days 1 hours ago saying "msg kid51 commented on TT #905".
sjn there
thanks, purl! :D
pmichaud can anyone think of a way to remove an exception handler (not necessarily the last) from the current list of active exception handlers? 13:10
13:12 rurban_ joined 13:15 rurban left, rurban_ is now known as rurban 13:18 Patterner left, Psyche^ joined, Psyche^ is now known as Patterner 13:23 whiteknight joined
whiteknight good morning, #parrot 13:26
yay! messages! 13:27
atrodo It's always exciting to discover that someone wants you 13:30
luben_work plobsing, ping 13:39
dalek ast: d94794a | moritz++ | S32-io/note.t:
unfudge .note test for rakudo
 13:49
moritz are you interested in the commits from roast? if not I can probably teach dalek to only report them to #perl6 13:50
dalek kudo: a15b2b1 | moritz++ | src/core/Bool.pm:
implement Bool.pick
kudo: 52bf6f3 | moritz++ | docs/ChangeLog:
update ChangeLog
14:01 fperrad joined
dalek kudo: 15c3f75 | moritz++ | src/core/Bool.pm:
remove Cool type constraint from Bool.pick

It doesn't allow a Whatever-star, so it's certainly wrong. It now let the re-dispatch to Parcel.pick do the type check or coercion, if any.		 14:03
14:08 aloha left 14:09 bacek left, uniejo left 14:10 bluescreen left
NotFound pmichaud: ping 14:11
pmichaud NotFound: pong
14:11 aloha joined
NotFound pmichaud: I've ruminated several times if that removal will be useful. There is no current way, AFAIK 14:11
pmichaud NotFound: okay. 14:12
I've since decided it's not the right way to go for what I'm working on.
NotFound Solved my doubt, then ;)
plobsing luben_work: pong 14:14
NotFound t/pmc/boolean.t Failed test: -- 23 not ok 23 - 0xor1 == 1 for Booleans 14:18
dalek ast: d71f4c3 | moritz++ | S02-builtin_data_types/bool.t:
Bool.pick
14:20 bacek joined 14:23 fperrad left 14:24 fperrad joined, bluescreen joined
dalek rrot: r49017 | gerd++ | trunk/NEWS:
some news for 2.8.0
 14:25
pmichaud what I'm looking for is a way to invoke a block from an exception handler with the current handler disabled for the duration of that block. 14:26
oops, I'll rephrase into Parrot terms
what I'm looking for is a way for an exception handler to invoke a Parrot Sub such that any exceptions thrown by that Sub won't be caught by the handler 14:27
NotFound pmichaud: let it propagate to the next handler?
14:27 contingencyplan joined
pmichaud I don't understand that phrase 14:28
NotFound pmichaud: if an exception happens, let the next handler in the chain, if any, handle it.
pmichaud how do I do that?
NotFound It was a question.
Is that the behaviour you want?
pmichaud if an exception happens for the sub the handler invoked, then yes, let it simply proceed to the next handler. 14:29
NotFound Let me look at it...
luben_work plobsing, I am working on deprecations of logical vtables 14:32
NotFound Hackish way that may be useful for testing: set min_severity on the EH to EXCEPT_exit + 1
14:33 Andy joined
pmichaud ...and then restore it when we get back from the Sub? 14:33
NotFound pmichaud: yeah
plobsing luben_work: cool. anything you need help with?
luben_work TT #1655. Yes
pmichaud I'm wondering if that means it's possible for it to not be restored, though. 14:34
luben_work I have removed vtables and have updated logical ops on PMCs to use get_bool/set_bool
NotFound pmichaud: the looking of the code is not much sphisticated, I think this simple approach should work well enough to test the idea.
luben_work the ticket suggests that we should remove the ops also. But I have not found a way to implement same functionality in pure PIR 14:35
pmichaud yeah, it might be sufficient to at least see what else might break.
NotFound pmichaud: if it serves your needs, we can add a 'disabled' attribute or something like that. 14:36
luben_work as s start, I do not how to call VTABLE_get_bool on any PMC (regardles of its type)
pmichaud there's already an unused 'invoked' attribute.
plobsing luben_work: it should be possible. don't we have some kind of get_bool op?
luben_work plobsing, no
I have double checked\\
pmichaud (fossil from previous mechanism used to disable exception handlers)
NotFound luben_work: if
pmichaud istrue
$I0 = istrue $P0
(invokes the get_bool on $P0) 14:37
NotFound What we don't have is a way to call VTABLE_set_bool from pir
luben_work NotFound, yes, that also is a problem 14:38
istrue will work for get_bool
NotFound pmichaud: the one in Continuation, you mean? 14:39
pmichaud NotFound: yes. ExceptionHandler overloads get_integer and set_integer to twiddle that flag.
14:40 timbunce joined
plobsing set_bool should be added as it seems to be an omission 14:40
pmichaud but I think those overloads exist only to preserve backwards compability somewhere, not because they're actually ued.
*used
luben_work plobsing, ok, I will work in that direction 14:42
NotFound plobsing: You mean an opcode?
14:43 timbunce left
plobsing NotFound: yes. if we have istrue, it is only logical that we be able to set it. 14:43
14:43 timbunce joined
timbunce what's the url for the parrot testing smolder instance? 14:44
NotFound plobsing: other than ortogonality, there is some reason that pays the price of an specific opcode?
moritz aloha: smolder? 14:45
purl smolder is sourceforge.net/projects/smolder or web-based smoke test aggregator used by developers and testers to upload (automated or manually) and view smoke/regression tests using the Test Anything Protocol (TAP). or smolder.plusthree.com/app/public_pr..._reports/8
14:45 fperrad left
aloha moritz: I have no idea. 14:45
pmichaud ...is there a way to get the pmc of the current exception handler from within the handler itself?
plobsing NotFound: there doesn't seem to be any other way of calling set_bool vtable from PIR 14:46
14:46 fperrad joined
plobsing seems like something PIR should be able to do 14:46
NotFound pmichaud: last time I looked at it, no. finalize accepts both the excepcion and the handler because of that. 14:47
plobsing: looks like people has survived a log time without tha hability. Just for testing purposes is not enough, IMO. 14:48
pmichaud I agree.
NotFound I don't like walking in circles, and I'm almost sure that in short time someone will suggest deletring it bcacues it's useless. 14:49
pmichaud and the only pmcs that appear to implement set_bool vtable are Integer, Float, and String.
(and boolean)
although it does get called from a variety of places.
14:54 theory joined
NotFound pmichaud: ah, yes, the handler_iter attribute in the Exception 14:56
pmichaud NotFound: Perfect.
purl perfect is the enemy of good enough.
pmichaud NotFound++
NotFound Look a t finalize in experimetal.ops 14:57
pmichaud eh = VTABLE_get_pmc_keyed_int(interp, iter, -1); 14:58
...is that safe? ;-)
i.e., I can be guaranteed that iter[-1] is always the previous thing iterated? 14:59
NotFound Safe enough to make finalize work
pmichaud I mean in the sense of "is this a valid public API I can rely on?"
NotFound Given that finalize is experimental, hardly a public API.
But if we want it to be a public API we just need to declare it as such and write some tests. 15:00
pmichaud I'd rather have a method on Exception to return its current handler 15:01
NotFound That feature, I mean, not finalize.
pmichaud right.
put another way, perhaps the current handler should be an attr of Exception PMC 15:02
NotFound pmichaud: I think a method is better. Public attributes exposes implementation details that maybe we need to change later. 15:03
pmichaud I'd be okay with that... although methods are currently very slow.
I only recommend attribute because the other attributes are already exeposed in Exception PMC 15:04
NotFound The usage you have in mind is speed critical?
pmichaud no, not terribly.
which is why I'd be okay with a method. :)
NotFound Then we can go for the method, and provide a faster way later with usage data if we need to. 15:05
s/with/based on 15:06
15:11 timbunce left
nopaste "plobsing" at 192.168.1.3 pasted "sources of empty hashes in rakduo" (16 lines) at nopaste.snit.ch/23357 15:11
NotFound Uh, pod in Exception.pmc is severily outdated.
15:11 masak joined
NotFound "When an exception handler is called, the exception object is passed as as the first argument, the message as the second argument of the call" 15:12
plobsing where are all of those hashes in bytecode comming from?
masak s/as as/as/ 15:13
NotFound Someone has worked on trac.parrot.org/parrot/ticket/1406 or is just a good wish? 15:15
15:16 SingAlong joined
NotFound plobsing: I bet ParrotInterpreter and Namespace provide a bunch of them. 15:17
plobsing I dind't think namespaces were frozen to vtables 15:18
s/vtables/packfiles/
SingAlong k. just got back to punch thru the PCT Tutorial again and reproduce my earlier problem
15:18 Coke joined
NotFound plobsing: I never bet that parts of parrot don't do irreasonable things. 15:18
Coke msg bacek IWBNI aloha knew about private messages for seen.
purl Message for bacek stored.
aloha OK. I'll deliver the message.
15:19 Coke left
pmichaud std: rand() 15:19
p6eval std : OUTPUTĀ«[31m===[0mSORRY![31m===[0mā¤Unsupported use of rand(); in Perl 6 please use rand at /tmp/j38e0CHiZO line 1:ā¤------> [32mrand[33mā[31m()[0mā¤Parse failedā¤FAILED 00:01 114mā¤Ā»
15:21 nwellnhof joined 15:23 dmalcolm joined
nwellnhof t/pmc/boolean.t Failed test: 23 15:23
dalek rrot: r49018 | nwellnhof++ | trunk (2 files):
[str] Introduce growable strings
 15:34
rrot: r49019 | gerd++ | trunk/NEWS:
add and change NEWS for 2.8.0 release
luben_work nwellnhof, I will look at boolean.t failures
dalek kudo: 5507cf6 | masak++ | src/core/IO.pm:
[core/IO] created/modified/accessed/changed methods

Not spec yet, but the spec sounded very non-committal, and I need those methods now. :)		 15:35
luben_work concern about logical ops on PMC deprecation: If we remove them we could not make logical operations on Bool PMCs 15:45
dalek kudo: 323a672 | masak++ | docs/ChangeLog:
[ChangeLog] new IO methods
 15:47
15:47 chromatic joined
NotFound "Closed a lot of tickets" is a NEWS item? 15:50
Following that way, we can add an item: "Worked a lot" 15:51
whiteknight NotFound: we don't have a number of tickets closed yet, because we're not done closing them
once we have a number, we can make NEWS more specific 15:52
NotFound whiteknight: I don't think number of a tickets, opened, closed or watever, is an interesting NEWS item. 15:53
I think that r49018 is a way of saying: "We said strings are immutable? Hohoho." 15:54
whiteknight Yes, r49018 is a little suspect, but I think it can be made to work correctly 15:56
After all, immutable strings only guarantees that string headers point to buffers which appear to always be immutable. What we do to the underlying buffer can be more tricky 15:57
NotFound whiteknight: How on earth we can correctly mutate immutable things?
whiteknight think about it this way. We have a string A, which points to buffer "foo" 15:58
if we grow that buffer and add more stuff to it, we have buffer "foobar", but string A still points to the same place and has the same length, so string A still is "foo"
but now we can have string B pointing to the same buffer with length 6, and it says "foobar" 15:59
NotFound whiteknight: maybe that idea can be made to work, but at the cost of putting one more problem against multithreading. 16:00
whiteknight now think about the lower-level, inside the GC. we allocate a large pool and cut it up into buffers. any string might be directly next to any other string inside the pool
NotFound: that's the thing, the buffers still never change contents once they are allocated. But we can play a trick inside the allocator, and write new bytes at an area directly next to the buffer, and treat that as a new, larger buffer without ever changing the first string 16:01
NotFound I think you are just missing the times of several heisenbugs a week.
chromatic Suppose the initial growable string came from a Class's get_string and represents the name of the class.
whiteknight I don't like the term "growable" here 16:02
chromatic Suppose the initial makeablewrongable string came from a Class's get_string and represents the name of the class.
whiteknight we have a buffer in memory with fixed contents, which never moves.
if other strings choose to overlap their storage with that existing buffer, without ever modifying that "window", there's no change in semantics 16:03
chromatic ... until you want to ensure that the contents of a string never changes, at which point you sprinkle tests and copy operations throughout the system like holy water. 16:04
whiteknight when we concatinate, we can pretend to copy the original contents by simply pointing a new header at the same buffer, giving it a longer length, and writing more data in memory directly after the first buffer
I disagree. There are no tests or holywater necessary here 16:05
our strings are still immutable.
and immutable strings don't need to be copied around
chromatic If the contents of a string change, that string has mutated.
whiteknight I'm saying specifically that the contents do not change
A string is a header with a pointer and a length. If the pointer never changes, the length never changes, and the contents in that window never change, the string is immutable 16:06
nwellnhof The content never changes, there's only stuff appended to a buffer.
whiteknight if we happen to write new things in memory directly next to that original buffer, the original string is none the wiser 16:07
nwellnhof The only tricky thing is to figure out who is allowed to append to a buffer
whiteknight or how we guarantee that we have free space next to the buffer to append into
chromatic This relies on shared buffers? 16:08
nwellnhof yes, at least for the concat_s_s_s opcode
whiteknight what's wrong with shared buffers, if we guarantee that the contents are immutable?
nwellnhof i like shared buffers, too
whiteknight if I have two strings, "bar" and "foobarbaz", there's no reason we can't compact that down to 9 characters of total allocation 16:09
chromatic They have GC implications, but that's merely an implementation detail.
If this patch (which I'm starting to understand) merely shuffles buffer contents to avoid extra memory copying instead of changing the contents of strings, that's very different from my initial reading.
whiteknight I don't know what that patch does. I haven't read it closely enough. In theory though, if we end up sharing buffer space between overlapping immutable strings, that's not a problem 16:10
it's like the idea of a cheap substring, but we create them in a different order
if that patch doesn't do that kind of thing, it's probably wrong 16:11
NotFound Other than opinions about the idea, there is a proble with that code. You are modifying a, which is cost.
Yes, using a cast, but that just hides the problem.
You are lying to the compiler.
nwellnhof we have to use the const_cast thing with strings in many places. 16:12
NotFound If we are going that way we need to drop the const qualifier in all functions that take STRING* parameters.
nwellnhof: that is the problem, but incrementing it is not a solution.
We used to have that problem with PMC at one time. The times of lots of heisenbug everywhere. 16:14
chromatic We still have that problem with PMC and STRINGs because of the GC flags in their headers.
nwellnhof if we don't want the const casts, there's no other way than to change the function headers. 16:15
NotFound nwellnhof: at the risk of losing optimizations done by the compiler in favor of manual optimizations. 16:17
nwellnhof i have no problem with changing headers.
chromatic If this optimization works, it's probably more powerful than compiler optimizations. 16:18
NotFound chromatic: most string functions don't need to change PObj flags, unless we play tricks. The more tricks, the more the probem.
chromatic True.
NotFound Changing headers is a public API change.
Thus subject to depecation policy. 16:19
nwellnhof most of the other const cast ugliness is related to hash tables. 16:22
chromatic Lazily calculating hash values, right? 16:23
nwellnhof yes, that's one thing.
the other is that hash_put doesn't accept const values.
NotFound hashes are less problematic because direct acess to its internals is localized. But direct usage of the STRING content is everywhere, and we are providing macors to make it even more frequent. 16:24
The hashval of the string is another problem, yes. 16:26
My view in short: we can go the way of being creative with the string buffers, or we can take benefit of the imutability by using const evrywhere. But not bot things at the same time. 16:30
16:30 masak left
chromatic The biggest problem of string concatenation seems to be the recycling of unused STRING headers. 16:33
That's a GC problem.
NotFound If that is the case, attempts of optimizations in the buffers will have low impact. 16:34
chromatic They could have some. 16:35
16:43 tcurtis joined
whiteknight is there anything that we can do to avoid the churn of string headers like that? 16:45
16:45 fperrad left
chromatic Rewrite into StringBuilder. 16:45
whiteknight yeah, that's what I figured you would say 16:46
which is something I'm in favor of
chromatic If we had working escape analysis, we could recycle garbage more frequently.
nwellnhof we can't force every HLL to use StringBuilder. 16:47
whiteknight I personally would probably like to see STRINGs disappear entirely as a separate low-level type, and do everything through either String or StringBuilder
in fact, we would only need StringBuilder, and rename it "String"
NotFound I don't think we must optimize for cases that programmers will avoid an even compilers can handle.
pmichaud 16:33 <chromatic> The biggest problem of string concatenation seems to be the recycling of unused STRING headers. 16:48
16:33 <chromatic> That's a GC problem.
...given my test that showed that disabling GC made the string concatenation take longer overall, is it still really a 'gc' problem?
or is "memory allocation" being included as part of "GC"?
whiteknight pmichaud: that's likely due to the performance of the string compactor, which is decidedly part of the GC 16:49
chromatic Not just the string compactor, but marking every live GCable in the program repeatedly.
pmichaud if GC is disabled, we still mark?
chromatic Then sweeping every live GCable in the program repeatedly.
whiteknight chromatic: GC is off
chromatic No, but we do page.
and swap 16:50
NotFound I think this is one more argument against tricks with string buffers. If each string buffer is owned by just one string, deallocating it is trivial and does not need gc.
pmichaud the resulting string in my example isn't that big -- only 140K or so
whiteknight when the string pool fills up, we allocate a new, larger pool and copy all existing contents over
pmichaud I'm pretty sure I never hit swap
chromatic How many concatenations does that example perform?
pmichaud 25,000
whiteknight without GC to trim that pool down, we're carrying around a lot of cruft for each copy
pmichaud well, 50,000 I guess 16:51
chromatic Let's cut that 140k in half then multiply it by 25,000.
pmichaud rakudo: 70000*25000
purl 1750000000
p6eval rakudo 15c3f7: ( no output )
whiteknight ...and swap
pmichaud yeah, that's 1.7 gb there 16:52
chromatic All of those calls to malloc() add overhead that skew the benchmark.
NotFound With unshared buffers, one run of the gc can free all buffers owned by all temprary string headres discarded. 16:53
pmichaud interestingly, when I run no-gc and load Rakudo into the interpreter, it takes 10 seconds less than with gc enabled 16:54
basic program, no rakudo, with gc: 3 seconds
basic program, no rakudo, gc disabled: 14 seconds
chromatic There's nothing for the GC to free when loading Rakudo.
pmichaud basic program, with rakudo, gc disabled: 21 seconds
basic program, with rakudo, gc enabled: 30 seconds
right, understood that the 'mark' phase adds a lot of overhead when rakudo is loaded 16:55
nwellnhof pmichaud: i never got your test case in #1789 to run in anything near 3 seconds.
chromatic Not necessarily mark.
nwellnhof it always takes at least 15 seconds.
NotFound nwellnhof: buy a faster machine ;) 16:56
pmichaud anyway, I agree with the earlier discussion about r49018 -- it looks to me like it just reintroduced copy-on-write semantics. 16:58
nwellnhof i don't know the old COW string implementation, but i guess that r49018 reintroduces some of it. 17:00
NotFound Againg, I don't like walking in circles, 17:01
pmichaud perhaps I have a very biased view, but to me, it seems that the most common (unique) operations on strings tend to be concatenation, substring, and substitution. Those have to be fundamentally fast in Parrot. 17:04
NotFound pmichaud: but not necesarily for worst case scenarios unlikely to happen. 17:05
pmichaud repeated concatenations to a string are hardly "unlikely to happen"
NotFound pmichaud: systems with immutable strings have stringbuilders ot the like for some reason. 17:06
pmichaud NotFound: but not every language has a stringbuilder.
chromatic Parrot has a StringBuilder.
pmichaud you can say that Parrot only efficiently supports languages that do things in terms of stringbuilding, but that knocks out Perl 6.
NotFound pmichaud: is someone knows a way of having the best of both worlds, I'm all ears. 17:07
pmichaud NotFound: if immutable strings are fundamentally incompatible with Parrot's target languages, then I think immutable strings have to go. 17:08
I'm not saying they are incompatible.
whiteknight pmichaud: if you're using a String PMC, and that String PMC does efficient string building internally, does that matter to you?
pmichaud I'm saying that if it turns out that they are, then Parrot gets to pick which to support. 17:09
NotFound A possible solution, but costly, is to make the String PMC more complicated.
pmichaud whiteknight: Strings are largely opaque to me. I don't really care if they're in PMCs or string registers (more)
whiteknight that's my point exactly. Strings as a low-level type are not tenable really
Strings as PMCs allow us to have low-level details and abstract them away from the user
pmichaud the only reason I use string registers now is because nearly all of the string opcodes require them to be there. 17:10
certainly all of Rakudo and NQP's strings that get stored in lexicals are in String PMCs
(or stored as attributes)
dukeleto waves hello
chromatic Well yeah, that's the only way you can store them right now.
pmichaud Most of NQP and Rakudo's choice about how to handle strings is completely dictated by the API parrot provides, not by language design choice. 17:11
whiteknight if we ever want to have reliably good performance across the board, we can't be exposing our low-level string types to the user
strings are too much of an icky implementation detail for anybody outside parrot core to have to worry about 17:12
NotFound whiteknight: winxed uses native strings without any problem.
pmichaud NotFound: can winxed duplicate the sample PIR program I gave?
whiteknight NotFound: and if I write a Winxed program that concatinates lots of strings, your performance will go down too 17:13
and that's a stupid thing to be telling our end users
pmichaud whiteknight: yes, but that's not a function of the existence of string registers.
chromatic "If you write code that's slow, your code will be slow."
whiteknight "we provide concat, but if you're stupid enough to use it, everything goes to hell"
chromatic That's a fine thing to tell end users.
pmichaud It just means that the STRING type may be insufficiently advanced.
whiteknight STRING isn't really a type. It's a very low-level form of storing text 17:14
NotFound pmichaud: the one if the ticket 1789?
pmichaud NotFound: Yes.
whiteknight a PMC is the kind of object with an interface and an abstraction layer that people should be using
NotFound Mmm.. just tell me how to use rand for pir.
chromatic Hypothetical: if we deprecated every user-facing use of a STRING and replaced it with a String PMC, would that improve performance? 17:15
pmichaud $I0 = rand 10000 # get a random integer up to 10000
chromatic Bonus question: if not, why?
NotFound Oh, I was looking to the perl, sorry.
chromatic Super bonus question: if not, what's the *real* problem?
whiteknight chromatic: I believe it would, overall
pmichaud chromatic: if we simply replaced String register with String PMC, I wouldn't expect any performance improvement.
chromatic I invite you to do some profiling then, because I believe it would not.
pmichaud I can even prove it.
whiteknight because internally, String can act a hell of a lot like String builder, and concat costs drop way down
chromatic Not only in the program, but every PIR-facing use of STRING.
pmichaud whiteknight: STRING can act like String builder too. 17:16
whiteknight: if you can do it for a PMC, you can do it for String also.
whiteknight pmichaud: it really cant. STRING is too low-level
pmichaud *for STRING also
whiteknight STRING isn't an object type. Doesn't have methods. Has limited storage. Isn't subclassable
pmichaud I'm not claiming that one can replace String with STRING, though. 17:17
I don't see how that applies here.
dukeleto attempts to help OSUOSL folks figure out the smolder issue
cotto dukeleto++ 17:18
whiteknight waves a "dukeleto is #1 flag"
chromatic: If we got rid of externally-facing STRINGs, there's no real reason for String PMC to contain a STRING header as an attribute
we can store a pointer to the buffer directly. That's one less GCable per String PMC 17:19
chromatic Is the content of this ubiquitous String PMC mutable?
dalek ast: 8e4955e | KodiB++ | S02-builtin_data_types/ (2 files):
Added tests for Sets and Bags.
whiteknight The buffer itself is immutable. But we can repoint our pointer at different buffers if we need to
or we can make the PMC itself immutable if somebody wants that. Or we can have multiple String types that do whatever people want their strings to do 17:20
chromatic From the user point of view, if I have a String PMC in $P0 and do something with that PMC, say pass it to a function, will the contents of that String PMC remain the same *by guarantee enforced by Parrot itself* after the call?
whiteknight we do have (a lousy implementation of) readonly PMCs
chromatic: that's an implementation detail of the PMC type itself 17:21
chromatic Then get used to cloning PMCs.
whiteknight we can have *many* String-like PMC types
The caller can click the read-only flag on the PMC before passing it
chromatic Yeah, I fixed a handful of bugs like that with mutable STRINGs. 17:22
Do you know one reason why immutable STRINGs were so much faster than mutable STRINGs?
whiteknight I'm not engaging in a design discussion based on the fact that we've had bugs in previous implementations of things
NotFound real 0m2.289s
user 0m2.264s
sys 0m0.024s
whiteknight or that software occasionally contains bugs, regardless of the algorithm
chromatic You have to face the bugs inherent in a design decision. 17:23
whiteknight chromatic: it's sounding like, to me, that immutable strings are *slower* in some cases
chromatic Yes, and faster in many cases.
whiteknight if our users want to do a shittonne of concats, that's a very real performance consideration
chromatic If you are confident that you can audit the whole of your program as well as all of the C code that might possibly manipulate any of the values of your program, you can get away with flipping that flag only in the cases where you need it.
(if it is indeed as cheap as flipping a flag) 17:24
NotFound pmichaud: the time looks aproximate to the pir program. More testing is pointeless because the winxed generated code is almost identical to the pir.
pmichaud NotFound: right. Now try the winxed program after loading 'perl6.pbc' :-)
or, tell me how you would write the winxed program differently to avoid the repeated concats.
atrodo chromatic> That always seems odd to me. Every time any language has touted immutable strings (java/c#/javascript) people run funny hacks and extra magic to have reasonable speed for often changing strings 17:25
NotFound pmichaud: using StringBuilder
pmichaud (and then tell me how the p5 programmer should rewrite his program to be faster)
atrodo but I think that's implementation after seeing parrot did it
whiteknight immutable strings are great until somebody wants to mutate a string. We can't act all surprised and say "ur doin it wrong" every time somebody calls concat_s_s_
chromatic If you refuse to see the design tradeoff, you're going to come up with a lousy design. 17:26
whiteknight and if we don't want people using concat_s_s_s, take it off the table
I see the design tradeoff
chromatic I don't think you do.
pmichaud (and then tell me how the php programmer should rewrite his program to be faster)
NotFound That's my point. We can't switch to mutable to immutable to mutable... every time one problem of the current approach hits someone.
whiteknight if we want to fix the concat problem, we take concat_s_s_s away and replace it with concat_p_s_s, which always returns a StringBuilder
atrodo pmichaud> (use perl?)
pmichaud atrodo: :-) 17:27
atrodo: that is exactly what the bulk of users would end up doing :)
whiteknight and we have a concat_p_p_p, which always returns a StringBuilder
chromatic Okay, now someone answer the *root* question, which is "Why is SB faster than repeated concat_s_s_s?"
atrodo pmichaud> or if you're facebook, write a php->c translator... 17:28
whiteknight because StringBuilder isn't reallocating and copying data around on every operation
chromatic No.
nwellnhof SB saves some string headers
chromatic YES
whiteknight because StringBuilder contains magical fairies and unicorns
chromatic: we can get rid of STRING headers *entirely* 17:29
chromatic No.
NotFound Magical pony-sized unicorns.
whiteknight yes
pmichaud Can I concatenate a string with a StringBuilder?
if no, why not?
oh, it's inplace.
:(
chromatic You can remove the STRINGs from pmichaud's benchmark and demonstrate the same problem *without concat*.
whiteknight not to the same degree. concat is a special kind of drain 17:30
chromatic Try it.
pmichaud tries it.
whiteknight there are problems with strings throughout, but concat is especially lousy
chromatic Try it.
pmichaud removing the concats in my version makes the program run in under a second. 17:31
both with and without perl6.pbc loaded.
chromatic Replace all of the STRING manipulation with the single line $P0 = box 1.
Run it with and without loading perl6.pbc.
pmichaud without loading perl6.pbc, 0.030 sec 17:32
whiteknight so you're saying that because we have other performance problems, that string handling and concat in particular are not far slower than they should be?
pmichaud with loading perl6.pbc 0.730 sec
I must be misunderstanding the thing I'm supposed to change.
pmichaud nopastes
chromatic I'm saying that concat exposes the real problem. 17:33
concat is not the real problem.
pmichaud gist.github.com/581102 # replacing concats with 'box 1'
whiteknight concat is *a* real problem
we have many real problems
chromatic Run the profile, whiteknight.
Look at it.
The performance characteristics are the same *without concat* in this example. 17:34
pmichaud chromatic: that's not what I'm seeing.
nopaste "chromatic" at 192.168.1.3 pasted "no concat; same characteristics" (14 lines) at nopaste.snit.ch/23361
chromatic Real time without loading perl6.pbc: 0m0.017s 17:35
Real time with loading perl6.pbc: 0m1.834s
pmichaud yes, but how long does it take to load perl6.pbc ?
in the string concat example, it's negligable relative to the overall performance. In this example, it dominates.
whiteknight because in the concat example, the concats increase overall cost by several seconds, not several hundredths of a second 17:36
the PMC-only example is significantly cheaper than the string concat example
because string concat is expensive 17:37
chromatic Fine, whatever.
Remove STRINGs.
whiteknight We don't even need to remove strings. I would like to in the long run but that isn't really necessary for this one example 17:38
removing concat_s_s_s and replacing it with concat_p_s_s is sufficient here
changing String PMC so it contains a pointer to the buffer directly, instead of wrapping a STRING removes the need for one GCable per String PMC
pmichaud gist.github.com/581115 # reworking chromatic's example 17:39
there's another possibility, fwiw 17:40
chromatic I overwrite the previous string concat example.
17:40 Coke joined
pmichaud well, perhaps not. 17:40
Coke is there a failure in t/pmc/boolean.t? 17:41
pmichaud I was thinking that if StringBuilder PMC would support the concat p_p_sc opcode, then Rakudo could simply switch to using StringBuilder instead of String as its base string class
NotFound winxed version: string with concat: 0m2.183s StringBuilder with append_format: 0m0.124s
luben_work Coke, yes, I am wirking on it
NotFound Even with method call overhead and format parsing 17:42
pmichaud NotFound: I don't dispute that using StringBuilder is faster than String.
Coke luben_work: ok.
pmichaud NotFound: I'm saying that it's not a trivial task to get language XYZ to be able to optimize to use StringBuilder instead of String.
whiteknight if StringBuilder does not support the concat_p_p_sc opcode yet, it certainly should soon
pmichaud and we can't just tell all of the Perl/PHP/Python programmers out there that they have to rewrite their source to make it possible for the compiler to use StringBuilder instead of String. 17:43
whiteknight: I'm not sure that will help.
whiteknight This is why I really think String and StringBuilder should be merged together. The end user shouldn't have to care about performance characteristics so long as the interface stays the same
chromatic -1
purl -1
pmichaud Because with $P0 = concat $P1, $S2, $P0 has to be a new StringBuilder PMC -- we can't reuse the one in $P1
whiteknight pmichaud: it can be a new String PMC instead. 17:44
pmichaud whiteknight: and if I want to concatenate again, what do I do?
move it to a new StringBuilder PMC?
NotFound Is even possible to return a StringBuilder from the conctenation operation on String
17:44 Coke left
whiteknight the three-argument form always does create a new PMC for the result 17:44
the two-argument form can do it in place
that's nothing new 17:45
pmichaud I'm only using the three argument form, period.
Forget the two argument form exists -- it's completely irrelevant to Rakudo.
purl pmichaud, I didn't have anything matching two argument form exists -- it's completely irrelevant to rakudo
whiteknight what's the current behavior of the three-arg concat? it returns a new String, or returns the same String?
pmichaud returns a new one.
it has to.
whiteknight right
NotFound Sometimes purl seems a magical fairy 17:46
whiteknight if we get rid of StringBuilder, and add magical cheap concat to String, we can pass the string to concat and get back a new String from it
just like we do now, but internally it goes faster
pmichaud whiteknight: my point is that we can verify this without that work. 17:47
if you add magical cheap concat to StringBuilder (which we don't have now), then I can benchmark it without having to do anything with String.
if it works, then we can consider migrating StringBuilder's internals into String 17:48
whiteknight I can fix that tonight so StringBuilder works nicely with concat_p_p_sc
chromatic What happens when it doesn't run (much) faster? 17:49
pmichaud I can have the test program that uses StringBuilder in just a couple of minutes, once that's available.
how does StringBuilder do its concatenation operations now, ooc?
chromatic pmichaud, can you paste your concat benchmark again?
pmichaud it's in the ticket :)
but sure, I can paste.
chromatic Which ticket? 17:50
purl Which ticket are you referring to?
pmichaud 1789, I believe
whiteknight pmichaud: StringBuilder stores the STRINGS in an array, keeping a running tally of total size and best-fit encoding/charset
dalek rrot: r49020 | coke++ | trunk (3 files):
Fixup some makefile dependencies for chromatic++
chromatic Thanks.
pmichaud whiteknight: I meant what's the API for concatnenation
NotFound pmichaud: it grows a buffer by more or less doubling it each time gets full
rrot: r49021 | Paul C. Anagnostopoulos++ | trunk (4 files):
First round of improvements to Parrot debugger
pmichaud not "how does it do it internally"
whiteknight pmichaud: I don't know, I have to look at it.
I don't know why it wouldn't work already
17:51 cognominal joined
NotFound pmichaud: AFAIK it only knows push_string and the append_format method 17:51
Uh, no, it also has concatenate_str 17:52
pmichaud gist.github.com/581137 # what I get when trying to concat to a StringBuilder PMC
NotFound i_concatenate_str and i_concatenate
pmichaud those are the inplace (2-arg) forms
I need a 3-arg form.
whiteknight great. MMD problem. we'll get that sorted out tonight 17:53
pmichaud it's not really an MMD problem, it's a NYI problem I think :)
NotFound A three-arg form tha does not do a lot of copying will be tricky.
pmichaud correct -- the three-arg form still requires copying (which is why it's worth benchmarking before undertaking whiteknight's proposal) 17:54
however, in this case it should copy the array of STRINGs instead of the STRINGS themselves. 17:55
that could be faster.
NotFound Is not using an array.
Maybe it should.
pmichaud oh. (I was going by whiteknight's comment above that it did)
right, if it's a buffer.... we're back to copying. 17:56
whiteknight it may be a tree
I am not looking at the code right now
NotFound I think they talked about using an array of string during its implementation, but later switched.
whiteknight the exact details aren't really significant
pmichaud anyway, in all of this keep in mind that Perl 6's Str type is immutable 17:57
i.e., Perl 6 doesn't need mutable strings.
chromatic Has anyone else profiled this code?
pmichaud (other languages might, but neither Perl 6 nor NQP are requiring mutable string capabilities internally-- they tend to model strings as immutable also) 17:58
chromatic: I don't know if others have. I'd welcome confirmation or refutation of the timings on the ticket.
but so far it's been highly repeatable for me. 17:59
chromatic memcopy is 2% of runtime 18:00
(at least the memcopy within Parrot_str_concat)
Parrot_str_new_noinit() is a third of runtime.
(at least all of the calls from within Parrot_str_concat)
We have a few options here. 18:01
Let's discard what NotFound rightly characterized as a flipflop between mutable and immutable.
One option is to say "STRING concat is too slow" and try to fix that. 18:02
whiteknight I agree. We need to stick to our guns and keep immutable strings
chromatic One suboption is to try to fix it in place, such that users never have to rewrite the code that seems obvious.
Another option is to try to give them a fairly obvious idiom to which to rewrite their code using PMCs or SB or something.
18:02 theory left
NotFound I think we should test the approach of never sharing string buffers and freeing the buffer when the string header gets collected. 18:03
chromatic My preference is the third option, which addresses what I believe the profile shows is the real problem.
pmichaud NotFound: I think that's unimportant (more)
whiteknight I do not think Parrot should be in the business of providing or enforcing idioms. We are trying to support too many separate languages and projects to do that.
pmichaud NotFound: one of the advantages of immutability is that it's okay to share data. 18:04
whiteknight people will reject Parrot on the basis that it uses foreign-looking idioms
NotFound Don't forget that if a language has particular needs for its string type it can just HLLmap whatever pleases it.
whiteknight that's one saving-grace that I see. If perl6 wants immutable strings, they can implement that. If Python wants mutable strings, they can hll_map those in 18:05
atrodo chromatic> I must have missed it, what is the underlying problem?
whiteknight or, mutable-looking ones (if internally the buffers are all immutable)
NotFound pmichaud: it's okay to share the strings, but not necesarily parts of strings. 18:06
whiteknight but chromatic is right on the one point: our performance woes are much larger than the problems with strings and concat
pmichaud NotFound: if substrings are going to become expensive, then we need a lot more work on Parrot's overall API.
chromatic The underlying problem is that Parrot_str_new_noinit causes the GC to run, which takes up 32.73% of runtime.
atrodo wow
pmichaud chromatic: is that problem still true when GC is disabled, ooc?
i.e., Parrot_str_new_noinit runs the GC even when it's disabled? 18:07
whiteknight no, it shouldnt
pmichaud (honest question, not rhetorical)
whiteknight but when GC is off, you run into several other performance-draining problems
chromatic Everything between Parrot_str_noinit (inclusive) and the GC is 0.28% of runtime.
dalek rrot: r49022 | luben++ | trunk/src/ops (2 files):
fix bug in logical not on PMCs when dest == src
whiteknight when GC is off, all that strain ends up on the string memory allocator, which is not sufficiently smart to operate without the GC in a performance-reasonable manner 18:08
pmichaud whiteknight: is the strain on the allocator itself, or on system resources?
whiteknight pmichaud: likely the allocator itself. It allocates large blocks which wouldn't cause too much fragmentation in the system
but you do end up then with lots of swapping and thrashing, which is a system issue 18:09
depends how you look at it
pmichaud well, I'm not hitting swap on my system.
whiteknight I haven't really looked at the performance of the allocator with GC turned off
but I can tell you it was certainly never designed for that situation 18:10
pmichaud I'm wondering if the allocator gets worse as there are more objects allocated.
whiteknight in terms of strings? yes
pmichaud if that's the case, then it's not just concatenation that is a problem.
whiteknight fixed-size objects like headers are allocated slab-style, so they should weather it a little better
pmichaud it's anything that reads or creates strings 18:11
whiteknight chromatic: are you looking at kcachegrind now?
chromatic yes
whiteknight what chunk of the performance pie does the string compactor eat up?
pmichaud I'm guessing from what chromatic reported earlier (0.28%) that the problem is really more in GC than allocation, though. not sure how that's being sliced.
afk, lunch
dukeleto can someone do "make smoke" with r49023 and tell me what happens?
chromatic Hard to say what the string compactor does, because I'm using my GC patch. 18:12
dukeleto i am getting stupid errors when trying to run "make smoke"
chromatic With -G, memcopy becomes the most expensive part of Parrot_str_concat() -- 20% of runtime.
whiteknight If I remember correctly, the string allocator allocates a single large pool for all strings. When that pool fills up it allocates an even larger pool, copies everything from the first to the second, and frees the first 18:13
that may be an old and simplified explanation
NotFound Scary 18:14
whiteknight if the GC never runs, the pool fills up with all the little string parts, and also all the intermediate concat results
all that memory storage, plus the constant copying would create the memcopy numbers chromatic is seeing
NotFound Is really that bad? 18:15
chromatic I'm only reporting the memcopy directly within Parrot_str_concat.
whiteknight I have to look at the code.
I don't know that the string allocator uses memcopy, since it's expecting the pool to be fragmented after a GC run 18:16
chromatic Without loading perl6.pbc, the memcopy within Parrot_str_concat is 87.96% of runtime. 18:17
(with GC disabled)
dukeleto no ICU lib loaded
make: *** [t/op/testlib/test_strings.pbc] Error 1
anybody know what is up with that? 18:18
nwellnhof dukeleto: which subtest exactly?
dukeleto nwellnhof: not sure, i did a realclean and trying again. will let you know 18:19
nwellnhof: ./parrot -o t/op/testlib/test_strings.pbc t/op/testlib/test_strings.pir 18:20
18:22 luben_work left
chromatic I wonder what would happen if we switched small STRING content allocations to come from slabs, not that compactable mess. 18:23
whiteknight chromatic: I would love to see all string allocation happen in a more organized, slab-like manner
anything smaller than half a page should come from slabs
chromatic There's clearly a GC problem specific to STRINGs, but I still maintain the big problem here isn't specific to STRINGs. 18:24
dalek rrot: r49023 | dukeleto++ | trunk/lib/Parrot/Harness/Smoke.pm:
[harness] Update smolder submission info
whiteknight not to harp on the point, but we do have many problems. Calling any one the "big problem" sort of underplays the amoung of work we need to do
nwellnhof dukeleto: i'm on it 18:25
NotFound In the meantime, can you please revert r49018 ?
chromatic Any benchmark which creates and throws away a lot of garbage immediately will expose one really big problem.
dukeleto begs people to run "make smoke" again
whiteknight we have certainly never completely capitalized on the promise that immutable strings was supposed to deliver
dukeleto on trunk
chromatic Sure, but that's because we haven't finished improving the GC for immutable STRINGs. 18:26
18:26 luben_work joined
whiteknight encapsulating the string portion of the GC is the next item on bacek's gc_massacre tasklist 18:27
cotto dukeleto, smoking now
whiteknight once that encapsulation is complete and merged, we should be free to reimplement as necessary
chromatic That has to be our focus to land as soon after 2.8 as possible then.
NotFound We surely still have functions that return copies or clones "just in case"
whiteknight strong agreement
chromatic I removed the biggest just in case copiers.
The worst culprit was register copying in PCC. 18:28
NotFound cotto: me smoking and drinking 18:29
dukeleto 500 on "make smoke". arg. 18:30
18:31 luben_work left
nwellnhof dukeleto: the tests add in r48987 only work with ICU 18:31
dukeleto nwellnhof: can you fix that? 18:32
nwellnhof: or skip them properly if icu is not present ?
nwellnhof dukeleto: yes, should be easy.
dukeleto this concerns me: t/tools/parrot_debugger.t ................... skipped: parrot_debugger changes have rendered these tests obsolete. 18:33
NotFound dukeleto: blame the greek and me
dukeleto what the hell is the point of having a test file which is unconditionally skipped?
NotFound dukeleto: easy: if we don't have it, people will complain. 18:34
cotto no luck submitting the smolder report
500 error from the server 18:35
oh. You saw that already.
dukeleto cotto: thanks for playing. you get the "home" version of the game: a bag of coal. 18:36
cotto yay? 18:37
NotFound How do you make a link that points to a line in the irclog? 18:38
moritz NotFound: click on the timestamp on the left of the line 18:39
NotFound O, the time, I was looking for line number or something
moritz rakudo on latest parrot fails some Unicode tests 18:40
for example t/spec/S02-whitespace_and_comments/unicode-whitespace.t
NotFound moritz: after r49018? 18:44
moritz NotFound: I tested on r49023
it's bad there
NotFound moritz: and between charset massacre and r49018 worked? 18:45
moritz no idea
moritz really wished that parrot developers tested branch merges with rakudo
NotFound I guess ByteBuffer get_string change is the culprit.
moritz it's a much more complete test suite than parrot's test suite itself 18:46
dukeleto smolder.parrot.org/app/projects/report_details/3 18:49
SMOLDER IS BACK!
NotFound plays the Imperial March 18:50
dukeleto moritz: i have plans to test branch merges. we want to, we just need the infrastructure
dukeleto thinks it would be nice if people joined #osuosl on freenode and say thanks (the channel requires a registered nick)
moritz r49017 already shows the Unicode problems 18:52
nwellnhof dukeleto: the t/op/testlib issue is fixed 18:54
NotFound moritz: no need to perl suite, parrot tests are also failing 18:57
# src/gc/gc_ms.c:1284: failed assertion '!(*Buffer_bufflagsptr(str) & Buffer_shared_FLAG)'
dukeleto nwellnhof++
nwellnhof smolder.parrot.org/app/projects/tap_stream/3/72
dalek rrot: r49024 | nwellnhof++ | trunk (2 files):
Skip tests from r48987 without ICU
 18:59
cotto Could we get a buildbot hosted somewhere where the bus number is greater than 1?
ttbot is great, but it's been a while since mj41 has been around and we'll eventually want the bot to pull from git instead of svn. 19:01
nwellnhof the gc_ms.c assertion is caused by r49018
NotFound nwellnhof: I was almost sure, but checking r49017 just in case 19:02
Yes, it pass with r49017
nwellnhof Parrot_str_escape_truncate reallocates a string from Parrot_str_concat 19:03
which might be shared
moritz r49005 is good
NotFound Urgh. That function still exists and is used?
Ah, no, I was thinking that it worked in place... but still ugly. 19:07
whiteknight dukeleto++ 19:09
whiteknight is going to have to run a lot of smolder reports tonight, to make up for all the ones I've missed with the server down 19:10
NotFound set S0, utf8:"\\u2001\\u2002\\u2003\\u2004\\x{e01ef}\\u0114"
\\u2001\\u2002\\u2003\\u2004\\x{e01ef}\\u0114
WTF?
whiteknight dukeleto: Are we able to add other projects in there? 19:13
dukeleto whiteknight: i can try. what do you want?
whiteknight dukeleto: first thing that comes to my mind is PLA. I would love to be able to push smoke reports somewhere for PLA
I think any other ecosystem project which is actively maintained and developed could make good use of the service 19:14
plumage might also be a nice one. partcl 19:15
(does rakudo have their own test reporting service?)
dukeleto whiteknight: rakudo does not, to my knowledge
moritz they have 19:16
had
tinyurl.com/rakudosmoke
NotFound Oh, shit, is right.
whiteknight I don't know what the limitations are for our server from OSUOSL, so we might not be able to handle reports from high-bandwidth projects like rakudo 19:17
(if we can, I'm happy to offer them the option)
dukeleto whiteknight: smolder.parrot.org/app/projects/smoke_reports/2 19:19
whiteknight: see Parrot::Harness::Smoke for how to submit to it
whiteknight thanks! 19:22
NotFound cmp.ops 1000:inline op xor(invar PMC, invar PMC, invar PMC) :base_core { 19:27
Shouldn't the first arg be out?
pmichaud 18:25 <whiteknight> we have certainly never completely capitalized on the promise that immutable strings was supposed to deliver 19:28
fwiw, I did a benchmark run of x1.pir against older versions of parrot. The newer versions are significantly faster in this respect. :)
so, not disagreeing with whiteknight++, but wanted to note that strings *have* gotten a lot better over time. 19:29
NotFound Until now? 19:30
dukeleto anybody want smolder projects?
whiteknight pmichaud: I'm not happy with some improvement. we really need ground-breaking, jaw-dropping performance improvements
dukeleto++
dukeleto i can make one right now, for whoever wants one.
i made one for plumage and parrot-linear-algebra, so far
and plparrot 19:31
purl plparrot is the postgres+parrot integration project or github.com/leto/plparrot
pmichaud whiteknight: I'm definitely in favor of ground-breaking, jaw-dropping performance improvements. :)
tcurtis /me runs "svn up; make reconfig && make && make smoke" 19:32
cotto but if they were easy...
NotFound pmichaud: that's easy, just insert an abort in imcc_main
An all benchmarks can run in less than a milisecond.
pmichaud "We fail a lot, but we do it faster than anyone else!" 19:33
dukeleto tcurtis: i do make realclean before, to be sure. reconfig doesn't work all the time
moritz "getting the wrong result in O(1) is easy" -- MJD
NotFound pmichaud: good slogan
cotto mjd++ 19:34
mj41 cotto: Good evening from Czech republic.
tcurtis dukeleto: reconfig does realclean first.
nopaste "chromatic" at 192.168.1.3 pasted "significant improvement in STRING benchmark" (49 lines) at nopaste.snit.ch/23363 19:35
dukeleto tcurtis: hmm. well then.
moritz ouch. My bisect identified r49014 as causing the Unicode problems in Rakudo.
cotto hi mj41
our wiki seems to be down 19:36
moritz dukeleto: could we please have a smolder target for rakudo?
dukeleto moritz: of course! that should be an interesting stress test....
moritz dukeleto: I thought ours also worked, but it turns out it didn't
NotFound Guru Meditation:
XID: 441151254
whiteknight did trac go down?
NotFound parrot.org runs in a Commodore Amiga?
I mean, trac.parrot.org 19:37
cotto mj41, what would it take to get ttbot to pull from a github repo instead of svn?
same for parrot.org
whiteknight smolder is also down
purl okay, whiteknight.
whiteknight damnit purl
purl i think damnit i am a bot!!!
tcurtis smolder? 19:38
purl hmmm... smolder is sourceforge.net/projects/smolder or web-based smoke test aggregator used by developers and testers to upload (automated or manually) and view smoke/regression tests using the Test Anything Protocol (TAP). or smolder.plusthree.com/app/public_pr..._reports/8 or down
NotFound Use some better machine, like an Atari ST for example.
dukeleto is talking to OSUOSL
whiteknight stress test preliminary results: FAIL
dukeleto BACK 19:39
whiteknight yes it is. Why did it go down?
dukeleto i was asking the OSUOSL folks to try and get https for smolder, so our smolder admin pass wasn't going over plaintext
GeJ Bonjour everyone.
whiteknight hello GeJ
dukeleto perhaps https for smolder is not worth it 19:40
mj41 cotto: taptinder clients are driven by taptinder server, which has database data copied updated from svn repos. So it is not so easy. It's first item on my todo list, since bacek make donation to taptinder git support.
GeJ Hi whiteknight. 19:41
NotFound Isn't PBC_COMPAT invalid? 19:42
# please insert tab separated entries at the top of the list
whiteknight Figuring out how to make PLA's setup.npq and t/harness compatible with smolder submissions is a different issue entirely
NotFound Followed by a non-tab separated entry
Don't we have a test for that? 19:43
19:43 tcurtis left
NotFound (I guess the "please" is enough to grant correcteness) 19:44
whiteknight dukeleto++ 19:45
dukeleto evidently only one vhost per server can have ssl, and trac has it. so no ssl for smolder. oh well. 19:47
they are looking into it, but it is not high priority
moritz: smolder.parrot.org/app/projects/smoke_reports/5 19:48
moritz dukeleto: thanks. What's the submission URL? 19:49
dukeleto moritz: all the details you need are in Parrot::Harness::Smoke 19:50
moritz: you just need to change your project_id to 5
moritz tries
dalek rrot: r49025 | mikehh++ | trunk/PBC_COMPAT:
PBC_COMPAT requires tab separators not spaces
rrot: r49026 | mikehh++ | trunk/src/debug.c:
fix codetest failure - trailing spaces
rrot: r49027 | mikehh++ | trunk/src/debug.c:
fix codetest failure - ASSERT_ARGS does not have a ; after and
mikehh damnit there was supposed to be a fix c_function docs after the and :-{ 19:52
on a second line but oh well
19:55 kid51 joined
kid51 dukeleto++ for getting our Smolder server up 19:56
particle++ for getting our Smolder server up
And my first submission to the new Smolder, smolder.parrot.org/app/projects/rep...details/5, is a FAIL!
t/op/string_cs.t test 49 19:57
kid51 goes back to $job
dukeleto we need one of the irc bots to look for fails on smolder and post them here. that shouldn't be hard 19:59
TimToady phone 20:00
20:02 whiteknight left
dalek rrot: r49028 | nwellnhof++ | trunk/src/string/api.c:
[str] Don't use str_concat in Parrot_str_escape_truncate
 20:08
20:26 tadzik joined 20:29 theory joined, Andy left
dalek ast: ea8f32d | KodiB++ | S02-builtin_data_types/bag.t:
[bag.t] Use .roll instead of .pick with :replace.
 20:44
mikehh All tests PASS (pre/post-config, make corevm/make coretest, smoke (#8), fulltest) at r49028 - Ubuntu 10.10 beta {g++-4.5 with --optimize)
the test t/op/t/op/string_cs.t test 49 20:45
20:45 PacoLinux left
mikehh try again - the test t/op/string_cs.t test 49 is failing an assert args which does not happen with --optimized builds 20:46
so with an --optimized build it does not do the assert args, but get the appropriate result and passes 20:47
20:48 PacoLinux joined
mikehh s/get/gets/ 20:53
20:53 patspam left 20:55 whiteknight joined
kid51 mikehh: But we have to get that test to pass either way. I do not generally build with '--optimized', nor should I be required to do so. 21:00
mikehh kid51: yes I agree, just explaining why it passed for me - am looking at it 21:03
21:03 davidfetter joined
nwellnhof t/op/string_cs.t should be fixed in r49028 21:04
mikehh kid51: I usually do an optimized build when I am going to test rakudo and others
bacek good morning, humans 21:07
chromatic bacek
bacek chromatic, hi
chromatic Have you left the high tech world of the future for the sheep-farming world of Australia yet? 21:08
bacek chromatic, even worse... I was quite busy at $dayjob and didn't have any energy to do coding. 21:09
"Powerpoint Architect", sigh...
cotto yech
chromatic I've rounded up several suspects willing to enforce your will on gc_massacre to get it to a mergeable state after 2.8.
If you let us know what to do, we'll do it. 21:10
bacek chromatic, it's blocked on proper String GC encapsulation.
chromatic Do you have a list of what we need to do there?
or at least a sketch of what you'd like to see?
bacek 1. Decouple string pools from old GC.
2. ... 21:11
3. Profit
Check compact_pool.
It's poking into header_pools of old GC.
chromatic Does it need its own pools?
bacek Nope.
It need something like GC_Subsystem->iterate_over_live_strings(callback) 21:12
chromatic Oh, it hardcodes the pools.
bacek Aye.
21:12 rurban_ joined
chromatic What if we moved all of the "You may compact string storage pools now!" logic into a single function pointer each GC can provide? 21:13
bacek I don't like it.
chromatic GC_Subsystem->sweep_pmc_pool
bacek I prefer "String GC" as separate subsystem. 21:14
chromatic GC_Subsystem->sweep_string_pool
bacek chromatic, it's other way round
sorear bacek: FWIW, if you were to reimplement utf32 and utf16 portably, NQP-rx would stop depending on shared buffers
bacek For compacting we have to iterate over _live_ objects
cotto Why are strings special wrt gc? 21:15
chromatic Sure, and if we keep the PObj_lives flag in STRING headers, we have to iterate over live objects too.
21:15 rurban left
bacek chromatic, we are compacting live strings. 21:15
21:15 rurban_ is now known as rurban
bacek or we can introduce refcount for string buffers. 21:16
21:16 dduncan joined
chromatic I agree; I don't see any disagreement. 21:16
21:17 kid51 left
bacek Then after sweep we can just iterate of buffers. 21:17
NotFound sorear: shared buffers for utf? Why?
21:17 dduncan left
mikehh nwellnhof, kid51: t/op/string_cs.t passes at r49028 with out --optimize on amd64 gcc-4.5 21:17
bacek No, we can't. Variable_Size_Pool doesn't store buffer boundaries.
chromatic Separate sweep and compact stages?
bacek chromatic, they are pretty much separated already. Just abstraction leak between "Headers GC" and "String Buffers GC" 21:18
... which have to be fixed.
chromatic I don't see how to separate "sweep string headers" from "compact buffers".
bacek After "sweep" we have only live string headers. 21:19
GC can provide iterate_string_headers(callback) which will be used by compact_pool
21:20 theory left
chromatic How do we iterate over only the live headers? 21:20
bacek In GC MS2 they are stored in simple list.
chromatic Are there any gaps in the list?
bacek sweep just remove dead objects from this list
chromatic, nope
chromatic Okay, that was the part I didn't understand. 21:21
bacek gc_ms2_sweep_pool
21:21 theory joined
bacek line 977, 1000 in src/gc/gc_ms2.c 21:21
21:21 theory left
chromatic You need someone to add that function and make it work. 21:21
Not just for GC MS2 but for MS1 21:22
bacek Yes.
It can be done.
chromatic ping luben, nwellnhof, whiteknight
whiteknight pong
chromatic We have a next task for gc_massacre. 21:23
cotto smells like progress 21:24
chromatic That's going to make GC MS1 look worse from a performance POV. 21:25
bacek cotto, can you retest gc_massacre? I changed gc triggering logic and it shouldn't eat all memory now.
chromatic, why? 21:26
cotto I'll do that. 21:27
chromatic GC MS1's compact_pool walks the string header pools to flip the PObj_live flag and compact at the same time, right?
bacek It's just one more undirected call.
chromatic, nope. It's just compacting.
chromatic It's already making a second trip through the header pools?
bacek chromatic, yes.
all of them actually
chromatic That's not the design of a genius.
bacek not only string/buffers
chromatic PMC header pools too 21:28
?
21:28 bluescreen left
bacek for (j = (INTVAL)mem_pools->num_sized - 1; j >= 0; --j) { 21:28
Fixed_Size_Pool * const header_pool = mem_pools->sized_header_pools[j];
_all_ of them
chromatic The good news is that fixing that should be a cheap way to get a couple of percentage points of improvement. 21:29
The bad news is that I need to repaint my office wall.
cotto build looks good; testing now 21:31
sorear NotFound: NQP-rx uses substr($orig, $pos, infty) to implement string cursors 21:33
NotFound: NQP-rx needs string cursors because Parrot Win32 doesn't have any Unicode string encoding with O(1) indexind 21:34
if Parrot gained a portable implemententation of utf32, or Parrot went back to supporting ICU unconditionally, NQP wouldn't need the substr hack
NotFound string cursors?
sorear iterating over a string in order to match bits of it against a regex 21:35
NotFound sorear: can't it use a StringIterator?
sorear that sounds like a question for pmichaud 21:36
probably it avoids stringiterator because of pmc overhead
it looks like our StringIterator uses indexing inside :( 21:37
cotto bacek, some test failures but the run completes without killing my machine 21:38
sorear so it won't do
bacek cotto, test failure are expected.
NotFound sorear: ATTR String_iter iter; /* String iterator */
StringIterator uses a String_iter
cotto ok. I won't bother nopasting them then.
sorear My copy of StringIterator doesn't have that line. *svn up* 21:39
the interesting case is when parsing large files, say core.pm, which is 157KB and has ~0.1% double-byte UTF8 characters
indexing near the end of that hurts
NotFound sorear: I think it changed in the charset masacre
whiteknight Can anybody enlighten me about how to post a smolder report using distutils? 21:41
the magic incantations escape me
sorear NotFound: definitely a pmichaud question, then 21:42
whiteknight dukeleto: ping
sorear will probably try this at some point
bacek $dayjob time. C U 21:43
pmichaud here's why StringIterator doesn't work. 21:48
Suppose I have a 30,000 character source code file.
dukeleto whiteknight: pong
pmichaud I've parsed through the first 29,000 characters.
I need to know if the next sequence of characters is the word 'whle'
'while'
...how do I do that?
whiteknight dukeleto: I'm having a hell of a time seding smolder reports
pmichaud (in PIR, please :) 21:49
dukeleto whiteknight: what is the problem?
purl hmmm... the problem is that [% user.comments.count %] makes TT call $user->comments in list context, which returns returns $user->comments_rs->all
21:49 hercynium joined, purl joined
whiteknight all the projects besides parrot dont allow anonymous uploads 21:49
nopaste "mikehh" at 192.168.1.3 pasted "test failures from make test in gc_massacre branch" (22 lines) at nopaste.snit.ch/23367 21:50
cotto forget the problem
purl cotto: I forgot problem
whiteknight it gives me the error about no anonymous uploads even when I am logged in as parrot-autobot 21:51
luben chromatic, pong 21:52
whiteknight when logged in as parrot-autobot, only the project "parrot" appears under the list of "my projects"
I suspect that user is not associated with all the rest of the new projects
chromatic luben, bacek gave some guidelines as to what we can do with gc_massacre. 21:59
pmichaud, is that due to a lack of ops which work on PMCs as opposed to STRINGs? 22:00
22:00 tadzik left
pmichaud chromatic: that's likely the biggest part of it, yes. 22:01
if we had string ops that could understand a StringIterator as an argument then it might be much better
but note that NQP/regex also needs the ability to backtrack
chromatic Right.
pmichaud i.e., if I iterate forward, I also have to have the ability to get back to where I was before
so the iterator can't just discard things
NotFound pmichaud: clone it 22:02
chromatic You almost need a PMC which can contain a STRING and a position within the STRING.
pmichaud NotFound: cloning gets *real* expensive very quickly.
sorear StringIterator has a pretty cheap .clone, it doesn't touch the underlying string
pmichaud that would be far more expensive than what we have now.
sorear (although it does make a new PMC)
pmichaud I'm pretty sure that substr is currently much cheaper than a PMC clone.
chromatic If iterating through the STRING via the PMC doesn't have to perform substrs, I can imagine that being much cheaper. 22:03
pmichaud well, if we want to implement all of the various string opcodes on our String-ish PMCs, that can work.
that would need to include find_cclass, find_not_cclass, is_cclass, etc. 22:04
NotFound You don't need ICU for that?
pmichaud we do need ICU for parts of it
sorear I think the ultimate fix would be to allow String_iter to be stored (opaquely) in an IREG, and then have ops working on that 22:05
(in practice it would be a byte offset)
NotFound sorear: And what black magic will distinguishit from a proper INTVAL?
dukeleto whiteknight: let me see if I can fix that 22:06
22:07 Paul_the_Greek joined
whiteknight dukeleto: on the parrot project, parrot-autobot is listed as a "Developer" 22:07
not so on the other projects
Paul_the_Greek Howdy, boys and girls.
sorear NotFound: PAST::Compiler type safety and a dash of Hungarian
whiteknight howdy dr nick
Paul_the_Greek ping nwellnhof
sorear NotFound: it's not like this is a memory address, it doesn't need to be interpreted by the GC's frame walker
dukeleto whiteknight: try now
nwellnhof paul: pong 22:08
whiteknight dukeleto: seems to work. I have access to the upload form no
now
I'm going to try posting a report
NotFound sorear: I think that way is too black magic and even more exposure of internals of a subsystem to completely unrelated parts.
Paul_the_Greek nwellnhof: Just a silly point: I presume that string catenation short-circuits the cases where one of the strings is the empty string.
dukeleto whiteknight: i just fixed that issue for all the smolder projects. thanks for noticing! ;) 22:09
nwellnhof yes, Parrot_str_concat does
Paul_the_Greek Just checking.
NotFound Looks like no.
purl hmmm... Looks like no. is there any reason I should not apply it?
Paul_the_Greek Concerning substring: Have you thought about preallocating all the ASCII 1-byte strings? 22:10
luben backloging now... I will look at the code tomorrow - it's after midnight here
Paul_the_Greek luben: Go to bed.
luben Indeed
NotFound Parrot_str_concat checks for STRINGNULL but I don't see any check for empty strings. 22:11
Paul_the_Greek What is STRINGNULL?
NotFound Paul_the_Greek: a null STRING
purl a null string is a hold over from C where a string is a char* and the addr that pointer references is 0
Paul_the_Greek That's what I meant by empty: ""
NotFound Paul_the_Greek: is not. 22:12
whiteknight dukeleto: Well, I just tried a smolder report again and it failed, but I suppose the error is on my side
Paul_the_Greek Is not?
purl Is too.
plobsing "" != STRINGNULL
Paul_the_Greek Now I'm confused.
dukeleto whiteknight: what error did you get? 22:13
NotFound Paul_the_Greek: we even have two kinds of null strings: STRINGNULL and (STRING*)NULL
22:13 theory joined
whiteknight dukeleto: no error. distutils only tells me "200 OK" 22:13
Paul_the_Greek What the hell? You have a preallocated empty (null) string, right?
plobsing kNotFound: not quite right. AFAIK, STRINGNULL = NULL for optimized builds. 22:14
Paul_the_Greek You only need one of them, after all.
NotFound Paul_the_Greek: yes, but in a lot of cases a NULL arg is allowed, so STRING_IS_NULL checks both
Paul_the_Greek Right, I can understand that, but what is STRINGNULL?
sorear A null pointer
NotFound Paul_the_Greek: an opaque object that means that the string is null.
chromatic It's a valid pointer.
plobsing a sometimes-valid pointer
sorear But fiddled to make it cheaper to test 22:15
Paul_the_Greek I'm roaring here. I'm completely confused.
More empty strings makes it easier to check?
NotFound Paul_the_Greek: welcome to parrot
sorear Paul_the_Greek: string->vtable->do_stuff
chromatic Are you familiar with the null object pattern?
sorear Paul_the_Greek: STRINGNULL has vtables that throw exceptions
NotFound If you aren't confused by parrot internals, you haven't seen enough of them yet.
sorear Paul_the_Greek: NULL is like STRINGNULL but instead of throwing exceptions it segfaults
Paul_the_Greek chromatic: No. 22:16
chromatic Okay, let me find a link. In the meantime, ignore everything sorear has written about this.
Paul_the_Greek So STRINGNULL represents the empty string and there are no normal strings of zero length?
NotFound STRINGNULL is a valid value of a S register, NULL is not.
dukeleto whiteknight: ug. perhaps try modifying Parrot::Harness::Smoke in parrot to submit to your project, just to see if it work?
works, rather
whiteknight: or perhaps try curl
Paul_the_Greek But why not just have a normal string of zero length?
chromatic Paul_the_Greek, see c2.com/cgi/wiki?NullObject
sorear Paul_the_Greek: STRINGNULL is not a string of any length 22:17
dukeleto whiteknight: i am not sure if the bug is distutils or smolder, at this point
NotFound Paul_the_Greek: STRINGNULL is a null string, not a zero length string.
plobsing Paul_the_Greek: we can have many 0-lengthed strings (one for each encoding) on top of stringnull
whiteknight dukeleto
purl dukeleto is probably mentoring a few peeps. can't remember everyone. sure. or jonathan at leeeeeeeeeeeeto dot net or hacking on git docs at github.com/parrot/parrot/tree/leto/git_docs
Paul_the_Greek What is the difference between a null string and a string of zero length?
whiteknight :I'll play with it. dinner time now
chromatic There is only one STRINGNULL in the system.
Paul_the_Greek plobsing: Yes, one for each encoding makes sense. 22:18
NotFound Paul_the_Greek: Have you used SQL?
chromatic It's a valid STRING pointer which means "This is not a valid STRING value."
sorear Paul_the_Greek: do you understand the difference between "" and (char*)NULL in C?
Paul_the_Greek Oh. Why do I want a string pointer that says it's not a valid string?
sorear: Yes.
Why not just use a universal NIL object for this? 22:19
Maybe we don't have a universal NIL object.
chromatic Because we want our C code to tell the difference between "A STRING pointer we know is not a valid STRING" and "Not a pointer."
NotFound We have several universals, we live in a multiverse %-)
Paul_the_Greek Okay, I guess I can understand that. I'll look for cases where NULLSTRING is used. 22:20
chromatic See also PMCNULL.
Paul_the_Greek Meanwhile, does it make sense to have one preallocated empty string?
PMCNULL I understand.
Oh wait, and so now I understand STRINGNULL!
NotFound Paul_the_Greek: we have: CONST_STRING(interp, "")
plobsing but that can only cover the ascii case 22:21
Paul_the_Greek And does every string function return that string when the result is empty?
NotFound He asked for one.
chromatic Meanwhile we should excise all uses of the NULL pointer as equivalent to STRINGNULL.
Paul_the_Greek Now, how about all the 1-character ASCII strings?
chromatic We don't perform any STRING coalescing.
Paul_the_Greek There immutable now, right?
s/There/They're/ 22:22
chromatic Right.
Paul_the_Greek So substr(, , 1) could return one of the preallocated strings.
NotFound Not a bad idea, provided we can have a way to obtain it from the interpreter quickly, 22:23
Paul_the_Greek Let's see ... 1,135 substr operations in *.pir
315 cases of substr , , 1 22:24
NotFound A lot less of string header cretaed is a lot of usages. 22:25
s/is/in
Paul_the_Greek Possibly 256 1-character ASCII strings and 128 1-character Unicode strings.
Or maybe just the first 128 ASCII characters.
NotFound Paul_the_Greek: there are no 256 ascii characters. 22:26
Paul_the_Greek There aren't?
plobsing Paul_the_Greek: do it in a branch. show that it is a worthwhile complication. 22:27
NotFound Wen we say ascii we mean ascii, not ascii-extended.
Paul_the_Greek Oh, 7-bit ASCII. Right.
plobsing: Not a bad idea.
Say, is there INTEGERNULL?
chromatic No, because integer and floats are value types. 22:28
NotFound Paul_the_Greek: is in the roadmap togethet with negative NaN
Paul_the_Greek What makes them value types and not strings?
dukeleto aloha, msg whiteknight bug is in distutils, i just submitted a PLA smoke report: smolder.parrot.org/app/projects/smoke_reports/2
aloha dukeleto: OK. I'll deliver the message.
dukeleto: Okay.
chromatic They're not pointers. 22:29
Paul_the_Greek Oh, they are immediate types instead of heaped types. Got it.
chromatic Do you have a Lisp background?
Paul_the_Greek Yes.
chromatic Okay, that makes it easier to explain then. 22:30
Paul_the_Greek I've worked on a few dynamic languages and I've never seen STRINGNULL, so that's why I was confused.
NotFound I think I'm starting to understand lexicals... should I be worried? 22:31
Paul_the_Greek NotFound: Closures? Be very afraid.
chromatic It's a little like NIL, except we don't have a singly-rooted hierarchy for heaped types.
NotFound Paul_the_Greek: lexical usage in pir, I mean
Paul_the_Greek Right, in Lisp we just used NIL everywhere we didn't have any other value to store.
NotFound: I've ignored it so far. 22:32
NotFound Paul_the_Greek: I can't, I need to make closures work in winxed. 22:33
Paul_the_Greek I'll buy you a beer once you're finished.
NotFound I have a test that works. Only for one level, but is a start. 22:34
nopaste "NotFound" at 192.168.1.3 pasted "Closures in winxed, first test" (31 lines) at nopaste.snit.ch/23369 22:37
whiteknight dukeleto: what was the bug? 22:38
dukeleto: and have you committed a fix?
dukeleto whiteknight: no bug. I just know that submitting reports for PLA works from Parrot::Harness::Smoke
whiteknight dukeleto: ah, okay
dukeleto whiteknight: either you are doing something wrong, or there is a distutils bug
whiteknight: sorry I wasn't clear about that 22:39
whiteknight that's okay
NotFound We got distracted from the initial point: str_concat do not special case zero length strings. Should do it, ignoring encoding? 22:46
chromatic Seems like a potential optimization.
NotFound I think at a time did it. 22:47
Not sure.
Paul_the_Greek It's a pretty simple optimization. 22:48
chromatic Easy to benchmark.
Paul_the_Greek Does fixed8_substr() still do ASCII substrings?
NotFound Paul_the_Greek: don't assume fixed8 is ascii. 22:49
Paul_the_Greek I've been trying to drill down to the actual substring function for ASCII and I'm going around in circles.
NotFound Before charset masacre fixed8 can be ascii, binary or iso-8859-1 22:50
Paul_the_Greek And after?
purl it has been said that after is like a BUILD by name, it seems.
NotFound Paul_the_Greek: after, it should not exist.
Paul_the_Greek What file contains substring in the massacre branch? 22:51
I think I have a catenate routine somewhere that cases on the length of the first operand and then, for each case, cases on the length of the second operand. 22:52
Does all sorts of cleverness.
NotFound Looks like fixed8_substr now is a function shared bt the appropiate encodings
Paul_the_Greek It copies the entire source string and then sets the offset and length.
NotFound So you can't use it unless you are in string internals and know what you are doing. 22:53
Paul_the_Greek: sounds difficult to adapt to a multiencoding world. 22:56
22:57 kid51 joined
Paul_the_Greek I think all encodings could special-case empty strings in catenation. ASCII could special-case 1-character strings in substring. 22:57
Oh, and empty strings, too. 22:58
I've also seen empty strings represented by an immediate value. STRINGEMPTY?
Probably not worth the special cases everywhere.
chromatic Probably not, as you have to be careful of encodings. 22:59
Paul_the_Greek Easier back when everyone agreed on only one encoding.
whiteknight msg fperrad Parrot has a new smolder server (smolder.parrot.org) and I can't find an option combination to make distutils send reports there. Any ideas?
purl Message for fperrad stored.
aloha OK. I'll deliver the message.
Paul_the_Greek Don't have to preallocate all the 1-character strings, but can allocate the single copy lazily. 23:01
chromatic I like that idea better.
Paul_the_Greek Vector of 128 pointers, lazily allocate, return allocated one. 23:02
Separate slot for pointer to empty string.
Have we considered allocating string headers and data together? 23:04
chromatic Makes walking the pools difficult. 23:05
Paul_the_Greek Allocate them in the fixed-size pools, for strings up to some size limit. 23:06
dukeleto RSS for failed smolder reports for parrot: smolder.parrot.org/app/projects/feed/1/failed
chromatic We still have to be able to identify which pool items are STRINGs. 23:07
NotFound Paul_the_Greek: very hard for a now, several parts of parrot still abuse string internals horribly.
Paul_the_Greek The could be in there own pile of fixed-sized pools, like PMCs are.
s/there/their/
NotFound Paul_the_Greek: take a look for example at windows specific functions in src/library,c 23:08
dalek ast: ea0bb44 | KodiB++ | S02-builtin_data_types/ (2 files):
Revamped keyhash.t.
 23:09
23:19 esskar left
bacek_at_work ~~ 23:42
23:44 Paul_the_Greek left 23:46 nwellnhof left
whiteknight blah. I have no idea why distutils is not posting my smolder reports 23:53
23:57 SingAlong_ joined

Powered by Cro and the RakuĀ® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!