01:08 pyrimidine joined 02:11 pyrimidine joined 02:15 KDr2 joined 02:24 pyrimidine joined 03:00 pyrimidine joined 03:04 pyrimidine joined 03:05 zakharyas joined 03:29 agentzh joined 04:06 pyrimidine joined 04:59 pyrimidine joined 05:05 pyrimidi_ joined 06:06 pyrimidine joined 06:33 pyrimidine joined 07:14 KDr2 left 08:36 pyrimidine joined 09:55 pyrimidine joined
jnthn moarning, #moarvm o/ 10:07
yoleaux2 5 Feb 2017 20:24Z <japhb> jnthn: Did you get my response to RT #130716? It isn't appearing for me on the RT site, but I don't know if my response got somehow routed to you and not included in the bug log, or if it just got lost entirely.
synopsebot6 Link: rt.perl.org/rt3//Public/Bug/Displa...?id=130716
yoleaux2 01:56Z <japhb> jnthn: Sheesh, it just took *several hours* to show up. WTH, RT?
dogbert17_ hello jnthn 10:13
I have some more comments wrt harness6 if you're interested 10:14
there are at least two problems with harness6, one was discussed the other day, running out of filehandles 10:15
the interesting thing with that is that this problem only occurs if TEST_JOBS=1, if set to 1+, the filehandle problem disappears 10:21
on the other hand, even though TEST_JOBS=1+ avoids the filehandle problem another problem plagues that setup, i.e. memory exhaustion at least on 32 bit 10:24
so running make spectest HARNESS_TYPE=6 [TEST_JOBS=1] fails due to the filehandle progrem while make spectest HARNESS_TYPE=6 [TEST_JOBS=1+] runs out of memory (2.5-3 gigs on my 32 bit machine) 10:29
jnthn Hmm, I wonder if TEST_JOBS=1 uses Proc rather than Proc::Async... 10:31
dogbert17_ for the first problem, see gist.github.com/dogbert17/069bf4c7...e3a1a9293a and for the other see gist.github.com/dogbert17/a74e36cd...6bcbc78931 which contains a tiny bit of valgrind output after having run a reduced spectest, i.e. ca 25 test files from S01 and S02 10:32
for problem two the amount of 'still reachable' memory grows alarmingly 10:33
samcv hi jnthn 10:57
10:58 pyrimidine joined
samcv .tell japhb that probably happened because you are not added to the rt list or something. send an email to That one was waiting in human moderation because you're not subscribed to the list. email [email@hidden.address] and they can add you 10:59
yoleaux2 samcv: I'll pass your message to japhb.
11:00 agentzh joined
samcv .tell japhb otherwise takes human moderation for it to go through 11:00
yoleaux2 samcv: I'll pass your message to japhb.
dogbert17_ jnthn: any ideas wrt the second gist?
jnthn dogbert17_: Well, it's the final leak source htat's the interseting one, I think 11:01
Though...does this not have --full-cleanup? 11:02
dogbert17_ nope
jnthn Hm, so hard to know if it's just poor collection timing or a real managed leak 11:03
Those buffers, I'm pretty sure, get wrapped up in a MVMArray
dogbert17_ well, when I ran a full spectest I got a MoarVM Panic, failed to allocate memory. I had run ps just before and VSZ was 3000000 and RSS 500000 11:04
I can run it again with full cleanup if you want 11:05
11:05 zakharyas joined 11:22 pyrimidine joined
samcv hmm jnthn ok so i'm trying to implement multi-level collation modes now 12:08
jnthn Sounds fun :)
samcv nqp::unicmp_s('a', 'b', 15, 0, 0)
collation_mode[0] lang_mode[48] country_mode[0]
:\
wat
jnthn o.O 12:09
samcv try it again and i get collation_mode[0] lang_mode[657072792] country_mode[0]
timotimo you're clearly reading uninitialized memory?
samcv they need to be MVMint64 maybe?
instead of 32?
timotimo must be the right thing in both ends
jnthn Or wrong register reading?
timotimo well, the same thing 12:10
all our int registers are 64bit
samcv ok then i will change it to MVMint64
timotimo smaller registers are 64bit big but we (try to) mask out the high bits whenever they are used
jnthn unicmp_s w(int64) r(str) r(str) r(int64) r(int64) r(int64)
samcv yeah that's what i guessed
i should stay with int64 even if i don't need 64 bit? 12:11
jnthn OP(unicmp_s):
GET_REG(cur_op, 0).i64 = MVM_unicode_string_compare(tc,
GET_REG(cur_op, 2).s, GET_REG(cur_op, 4).s);
samcv i only need like 1000 max for any of them i think
timotimo yeah, that'd make it easier
hah
jnthn Notice it doesn't even read/pass the last two args
timotimo you forgot to pass all the other values
jnthn To the function
So they end up as junk
samcv ah
timotimo and C be like "didn't pass any of those arguments? doesn't matter."
samcv lol.
jnthn heh, why on earth doesn't it warn though...
samcv don't care if you passed it or not! i'm gonna make up some for you!
some sane values 12:12
to just toss in there
timotimo well, i wouldn't notice a warning like that, because my terminal is always spammed with that diagnostic error
jnthn Well, they're integers, so that's fine, right? :)
samcv :P
timotimo well, calling convention is "just take arguments from the registers"
jnthn :)
timotimo and registers are always "initialized" :P
jnthn Time for lunch, bbiab
timotimo so it could have been that you put an inline assembly block before the call that sets the registers manually! 12:13
samcv what do i need to change it to jnthn
timotimo in that case, it would clearly be fine and totally what you meant and want!
samcv heh timotimo
timotimo well, put GET_REG(cur_op, 6).i64, GET_REG(cur_op, 8).i64, GET_REG(cur_op, 10).i64 into the parethesis
samcv kk
jnthn Add something like: GET_REG(cur_op, 6).i64, GET_REG(cur_op, 8).i64, GET_REG(cur_op, 10).i64
To pass the 3 extra args to the function call 12:14
Really off to eat :) bbi30
timotimo at least you're already incrementing the cur_op by the right amount
even though you didn't read any of the "late" values
samcv now i get too many 12:15
timotimo, gist.github.com/samcv/3632be9e3f90...3080740959 12:16
timotimo you only changed the function signature in the .c, not the .h? 12:17
samcv i didn't change the number in the c function for MVM_unicode_string_compare
it always had that many arguments
timotimo but you turned 32 into 64
samcv i changed the type tho. will check any .h
timotimo that could get you into trouble
but since you didn't get into trouble for not passing the right amount of parameters ... ?!? 12:18
samcv oh yeah it is too little in the .h
hehehe
timotimo you're supposed to get into trouble for having different stuff in the .h and in the .c
samcv there's no ints in the .h
timotimo but somehow you seem to get away with anything ever?
samcv is in trouble
basically
timotimo i suppose that's fair 12:19
though i was hoping for compile-time trouble
samcv jolly good. it compiled now
perfect :)
12:23 pyrimidine joined
samcv now to get it to work so it won't break ties if i unset quaternary 12:25
timotimo if it doesn't break ties, it just returns 0 if all else comes up as "equal"? 12:26
samcv yeah
that is one of the configurable things 12:27
quanternary is breaking ties by cp
quanternary is not defined by unicode other than 'whatever you want to do here'
timotimo that's fair 12:28
samcv tbh idk how many cases there would be where collation would come up the same
not likely i think
but it is more likely if you ignore some of the collation levels probably
like if you ignore diacritics (second level) 12:29
timotimo in the near future i'll never be able to trust that a sequence of strings that has been sorted and given to my program won't change when i try to sort it again inside my program
samcv you may want to unset quaternary so it won't break ties by cp
timotimo thanks unicode
samcv just don't set $*COLLATION and it should be predictable, or just use unicmp and not coll
or something
timotimo no, i mean
what if i get my list of strings from a python program?
samcv oh lol 12:30
timotimo or a database of some kind
samcv python sorts by cp probably right
idk. java doesn't sort by cp
timotimo no clue
samcv it sorts 'lexigraphically' which is undefined for the most part
12:30 harrow joined
timotimo :D 12:30
well, not completely undefined, but at least underdefined
samcv i thought that was hillarious
basically implying that don't trust it'll always be exactly the same
timotimo ugh 12:31
samcv though it'll always say whether they are equal or not, it may not have the same 1 or -1
who knows.
timotimo so at least it'll report same for one Ƥ and the other Ƥ
samcv will it?
timotimo well, i certainly hope so
samcv what are these two a's
timotimo those two Ƥs are actually the same Ƥ 12:32
samcv what are the two ones tho
timotimo because my keyboard won't let me easily input both variants
samcv what are their names
timotimo well, a + diaresis, and &auml;
samcv or you mean one with diacritic and one just its own character?
that has the :?
timotimo yeah, Ƥ has its own character, but you can also compose it 12:33
u: Ƥ
.u Ƥ
yoleaux2 U+00E4 LATIN SMALL LETTER A WITH DIAERESIS [Ll] (Ƥ)
samcv m: 'Ƥ'.NFD.say
camelia rakudo-moar 206148: OUTPUTĀ«NFD:0x<0061 0308>ā¤Ā»
timotimo funny that the html/xml entity name is "auml", but it's "diaresis" in the unicode name
samcv m: Uni.new(0x61, 0x308).Str.ords
camelia ( no output )
samcv m: Uni.new(0x61, 0x308).Str.ords.say
camelia rakudo-moar 206148: OUTPUTĀ«(228)ā¤Ā»
samcv well it recomposes
at least in perl 6
but if the case that it does not recompose, unicmp will show the same i believe 12:34
as long as unicode collation says so
which it probably does, or they are sorted very closely
timotimo in perl6 it certainly will
but what about python and others? :)
samcv heh 12:35
tell them to sort better
timotimo sort out your sort?
samcv yep
sort out your shit
timotimo shit your values in the right order?
12:35 pyrimidine joined
timotimo anyway. let's spend some big part of every day high-fiving ourselves because we'll now be able to sort emoji correctly 12:36
samcv hah 12:41
yes
^5
timotimo āµ
samcv ** 5 12:42
timotimo pow($_, 5)
samcv hah
12:43 pyrimidine joined 12:48 lucasb joined
jnthn back from lunch 12:49
lucasb can I hi5 too? yay
o/
samcv sweet got it working
yes you can lucasb
jnthn nice :)
samcv we all have to high 5 all day long
Geth MoarVM: ab225fa94d | (Samantha McVey)++ | 3 files
Implement configurable collation_mode for MVM_unicode_string_compare

This change causes the collation_mode argument to affect how the strings are compared. Quaternary level will ignore differences in codepoints and only rely on the collation values.
This also fixes a problem in interp.c where it did not pass the three integer arguments, and also fixes the arguments being incorrectly set to MVMint32 instead of MVMint64		 13:06
timotimo yes yes, very good
moritz we're comparing strings with Quaternions! 13:07
dogbert17_ jnthn: sry, was called away on a $meeting, will run valgrind with --full-cleanup now 13:11
jnthn dogbert17_: No hurry at all, I'm fairly tied up with other stuff today anyway 13:16
timotimo so ... you know how there's richard wagner's walkĆ¼renritt? wouldn't it be funny if someone put up a version of that slowed down to 0.05x and called it "valgrindritt" or something?
dogbert17_ lol 13:22
jnthn: you'll see some (shocking ?) results shortly 13:24
lucasb timotimo: the song that plays in that movie scene in vietnam with helicopters? :)
timotimo yeah
lucasb :D
timotimo apocalypse now, isn't it?
lucasb yes, that one 13:25
timotimo BBIAB 13:26
dogbert17_ jnthn, timotimo: dramatic results can be found here: gist.github.com/dogbert17/a74e36cd...6bcbc78931 13:36
14:37 pyrimidine joined 14:43 pyrimidine joined 15:01 agentzh joined 15:26 pyrimidine joined 15:46 lizmat joined 16:07 pyrimidine joined 17:09 pyrimidine joined 17:12 dogbert17 joined 18:00 pyrimidine joined 18:06 pyrimidine joined 18:21 pyrimidine joined 18:27 pyrimidine joined 18:53 lucasb joined 19:03 agentzh joined
lucasb What are the feelings of jnthn and the others about moar's documentation? any future plans? 19:07
Adding markdown files in the docs dir or creating a new doc repo would be considered useful/helpful?
TimToady it's all a trade secret! 19:13
lucasb oh :) 19:14
I was trying to learn moar by reading a little code and writing notes into markdown files. 19:28
Then I got sidetracked playing with the idea of generating a static site.
Most pages are empty and those that are not contains inacurate information from my notes: lucasbuchala.github.io/moredocs/
So I ask, is there any interest by the MoarVM community in adopting this documentation skeleton as a base for Moar's community doc repo?
the source of the site is a single directory with markdown files and some build scripts 19:29
20:06 pyrimidine joined
jnthn Documentation for MoarVM is kinda a question of "who is the audience" 20:28
20:30 lucasb joined
jnthn Tied for time, I've figured my best bet on docs so far is to try and stick comments explaining various key decisions/designs into the code. And then just spend some effort trying to make the code clear. 20:31
lucasb I see the audience as definitely not Perl 6 users; but anyone interested in Moar's internals, C source code 20:32
I noticed the very nice comments along the way, jnthn++, thanks for that
jnthn Yeah, the question is what (a) will help people in ways reading the code cannot, and (b) is likely to stand the test of time well enough to not become out-dated quickly and thus confuse people more than if it weren't there :) 20:33
I think there's plenty of such things that exist.
The MVMROOT macro and related is one example 20:34
Something on overall structure is another.
I'd probably be more inclined to link to appropriate bits of code than duplicate it in docs. 20:35
lucasb right, understood
jnthn Ah, gotta go to the store before it closes...back in 15-20 mins 20:36
lucasb The source is there for anyone interested in reading it. One could start reading it from start to end, or jump around until the person starts to retain the vocabulary of the code. Wouldn't it be nice to have a more "holding hands" experience of guiding through the code? 20:44
The intention is not to duplicate code, but just show the main pieces of it, and how they work. Sure, it can get outdated quickly, and what doesn't? It was just a try anyway :)
dogbert17 jnthn: I have some additional info wrt the problems with HARNESS_TYPE=6 21:01
if I run 'make spectest HARNESS_TYPE=6 TEST_JOBS=2' with an unchanged MoarVM, i.e. #define MVM_NURSERY_SIZE 4194304 it will run out of memory. Here's a 'ps' just before the crash 21:05
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dogbert 24198 37.1 16.0 2952656 497040 pts/2 Sl+ 19:50 7:32 /home/dogbert/repos/rakudo/install/bin/moar --execname=./perl6-m --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=. /home/dogbert/repos/rakudo/perl6.moarvm --nqp-lib=blib -Ilib t/harness6 --fudge --tests-from-file=t/spectest.data
if I change MVM_NURSERY_SIZE to, e.g., (32768 * 8) the program will NOT run out of memory. Here's a 'ps' just before it's done testing 21:07
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dogbert 24198 37.1 16.0 2952656 497040 pts/2 Sl+ 19:50 7:32 /home/dogbert/repos/rakudo/install/bin/moar --execname=./perl6-m --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=. /home/dogbert/repos/rakudo/perl6.moarvm --nqp-lib=blib -Ilib t/harness6 --fudge --tests-from-file=t/spectest.data
fail, pasted the first ps again it should be:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 21:08
dogbert 31864 44.7 5.6 516208 175928 pts/2 Sl+ 20:13 12:51 /home/dogbert/repos/rakudo/install/bin/moar --execname=./perl6-m --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=/home/dogbert/repos/rakudo/install/share/nqp/lib --libpath=. /home/dogbert/repos/rakudo/perl6.moarvm --nqp-lib=blib -Ilib t/harness6 --fudge --tests-from-file=t/spectest.data
does the above give you any clever ideas? 21:09
21:11 pyrimidine joined
jnthn lucasb: Yes, good point. I agree some kind of "map" would be useful. Something that explains what the different directories under src/ are about what be good and quite lasting. And some "reading tips" for each - both source code and background. For example, in src/spesh/ it's not obvious that graph.h is the most important file to start reading, and links to what a control flow graph, basic block, and SSA form are would be useful, for example. 21:16
dogbert17: Interesting. It suggests we're simply not reclaiming memory anything close to fast enough. 21:17
dogbert17 any theories as to why? 21:18
jnthn Various 21:19
dogbert17 :-)
jnthn Did you try playing with the gen2 collection thresholds here: github.com/MoarVM/MoarVM/blob/mast...llect.h#L6 21:20
dogbert17 no; i have only changed MVM_NURSERY_SIZE, never touched the others. Any suggestions?
for values I mean 21:21
jnthn Well, they control how often we do a full collect
The algorithm using them is in orchestrate.c
I think it should be relatively clear how it's doing that decision-making
But smaller means "will do gen2 sooner" 21:22
Well, full collection, which includes, gen2, really.
If that makes no difference then it suggests we might need some kind of memory pressure mechanism for the nursery
dogbert17 I'll do a quick experiment then, setting MVM_GC_GEN2_THRESHOLD_PERCENT 10 21:24
I'll set VM_GC_GEN2_THRESHOLD_MINIMUM to (10 * 1024 * 1024) as well 21:25
it worked with 10 instead of 20 21:54
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dogbert 16230 39.0 11.5 1279656 359588 pts/2 Sl+ 22:27 9:56 /home/dogbert/repos/rakudo/install/bin/moar ...
stupid question, how large can the value of 'promoted' be? 22:01
jnthn Well, it uses size_t in the calc, though it's stored as MVMuint32, so 4 gigabytes 22:08
If you mean maximum value
Or do you mean how large can it become between tests?
dogbert17 I was looking at this code 22:10
/* Otherwise, consider percentage of resident set size. */
if (uv_resident_set_memory(&rss) < 0 || rss == 0)
rss = 50 * 1024 * 1024;
percent_growth = (100 * promoted) / rss;
was wondering if promoted is somehow bounded in any way by some constant in the source 22:12
jnthn Oh
No, not other than the data type it's stored in
We add on to promoted every time we move an object from nursery to gen2 22:13
Both the object's body size, and then for objects that report it, any extra memory they hold indirectly that the GC won't otherwise know about
dogbert17 ok, in my runs rss is quite large so I guess it will take time before promoted is large enough to cause a full gc 22:14
or have I misunderstood /o\
jnthn No, you probably did 22:16
But
If there's a huge discrepancy between promoted and RSS then we might want to look into why
Uh, that ain't quite right 22:17
If there's a huge discrepancy between RSS growth and how big MoarVM thinks the heap is, then that's an issue.
As it means promoted is under-estimating.
dogbert17 how can we figure out if that is the case? 22:18
jnthn moar-ha
And --profile=heap
dogbert17 moar-ha ?
jnthn You can install the first with [zef|panda] App::MoarVM::HeapAnalyzer
moar-ha is the name of the "binary" it installs
That said, moar-ha itself slightly distorts collection rates 22:19
dogbert17 I saw a comment to that effect
what about these GCDEBUG_LOG statements in the code, might thay help and how do you see them? 22:20
jnthn Still, if there's a huge discrepancy between usual (non-profiled) RSS and the heap size...
Those are more usual for investigating a different class of bugs mostly
*more useful
For example if there are problems with inter-thread coordination of GC runs 22:21
dogbert17 aha
jnthn I've not needed them for quite some months, thankfully
(Because any problem I'm hunting with those turned on will typically be horrible.)
dogbert17 with some luck most of the horrible problems are gone now
I had better read up on the heap analyzer, didn't you blog about that? 22:22
jnthn Yes, the README in its github repo has a link to the blog post 22:26
dogbert17 will look into to it 22:27
jnthn Cool 22:28
dogbert17 might have gotten in over his head ...
jnthn One other thing to check; if you lower the threshold to 10%, what is the CORE.setting build time difference?
(I tuned the numbers using that)
dogbert17 I didn't check that 22:29
did you see the --full-cleanup gist I posted earlier? 22:32
i.e. gist.github.com/dogbert17/a74e36cd...6bcbc78931
jnthn Yes, it confused me some though 22:35
As it left behind things I'd not expect --full-cleanup to...
Like nurseries
dogbert17 is it because valgrind died, it says 'killed' on the last line 22:36
jnthn Oh, yes, it will be
m: say 60_489_728 / 923 22:37
camelia rakudo-moar 4efcc2: OUTPUTĀ«65536ā¤Ā»
jnthn That's a magic number :)
I thought that I'd already fixed procops.c to correctly track sizes though
dogbert17 maybe something erroneous has been mistakenly reintroduced 22:38
jnthn Mebbe, though it was a pretty recent patch... 22:49
Sleep time for me; 'night
timotimo nite jnthn 22:52
it would have been nice to know about the "killed" before :) 22:53
dogbert17 yup, my bad :( 22:54
timotimo, did you read about the latest findings above? 22:56
timotimo i read a little bit
we're promoting things that we don't calculate the correct size of, and therefor don't do a full gc? 22:57
dogbert17 that could be the case, running make spectest HARNESS_TYPE=6 TEST_JOBS=2 eats all memory and crashes, making the nursery smaller removes the problem 22:58
22:58 pyrimidine joined
timotimo interesting indeed 22:58
were you able to run it under the HA? 22:59
er, i mean, with the heap profiler output turned on
dogbert17 haven't tried it yet although I'm slighly sceptical, trying it on a oneliner is quite slow ...
timotimo it depends verily on how much gcing it does 23:00
each time it gcs, it records a full snapshot of the object graph (but leaves out anything but inter-object connections)
dogbert17 so should I make the nursery slightly smaller so that it barely avoids crashing or should I set it really small? 23:01
timotimo for the purpose of getting a moar-ha thing? 23:02
dogbert17 and to see if there's something fishy with the heap/rss/promoted
timotimo if you set it really small, it'll likely promote more stuff into gen2 23:03
dogbert17 yes, perhaps that is not what we want, with a normal nursery that doesn't seem ta happen too often since it runs out of memory 23:04
timotimo oh, i think if it crashes it won't write out the heap dump, i'm not sure about that 23:05
and it'll take a bunch of memory at the end to write out the heap snapshots
but also: it won't cross-reference system-reported RSS with the individual snapshots
maybe it'd be a better idea to just go through all the reprs and look at their managed_size functions? 23:06
dogbert17 ah, ofc, perhaps I should change spectest.data soo that it doesn't contain so many files, don't think we need to run through all of them
timotimo mhm
dogbert17 maybe it's enough to go through, let's say, 30 files in order to get good data 23:07
I'll read up on the heap analyzer and try it tomorrow, bedtime for me 23:09
night
timotimo nite 23:11
23:27 zostay_ joined 23:30 ilmari_ joined, konobi_ joined 23:40 timotimo joined

Powered by Cro and the RakuĀ® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!