github.com/moarvm/moarvm | IRC logs at colabti.org/irclogger/irclogger_logs/moarvm
Set by AlexDaniel on 12 June 2018.
00:15 patrickb left 00:57 scovit_ joined 01:02 scovit left 01:07 MasterDuke left 01:09 MasterDuke joined, MasterDuke left, MasterDuke joined
MasterDuke timotimo: fyi blog.mozilla.org/nnethercote/2019/...tter-dhat/ 01:35
04:50 ggoebel joined 05:14 domidumont joined 06:49 domidumont left 07:01 domidumont joined 07:29 brrt joined 07:51 patrickb joined 07:52 zakharyas joined
brrt \o 07:54
patrickb o/
jnthn o/ 09:14
timotimo o/ 09:24
afl found out that P6opaque just writes to whatever reprid slot is in the serialized blob at that position, no matter how high (but truncated to a 16 bit unsigned integer) 09:59
in its deserialize_repr_data function
huh. it doesn't write the reprid/slot if the reprid is 0, which would be MVMString 10:01
that can't be right?
Geth MoarVM: 275098a2db | (Timo Paulssen)++ | src/6model/reprs/P6opaque.c
fuzzing: don't accept out-of-range repr ids

also: accept 0 as repr id (it's MVMString)		 10:15
timotimo ==7491== Warning: set address range perms: large range [0x7a6d040, 0x17b574a8) (undefined) 10:16
what the? :D
10:22 domidumont left
timotimo i wonder if moar should actually immediately panic when it encounters a wrong value there 10:22
lizmat seems like a good thing to do to me 10:23
timotimo another piece of that function throws an exception when serialization goes wrong
which ... is kind of odd, but okay
also, the code should probably check against out-of-range slots, too
jnthn: is spesh robust against "log_parameter" just not doing anything in some instances? because afl found a way to smuggle a low-level null in there and it makes log_parameter's first line, which tries to get the container spec via STABLE(obj), segfault 10:47
jnthn timotimo: The problem is it got the low-level null in there in the first place, I'd say 10:48
We shouldn't solve problems by sprinking null checks everywhere, but rather figure out how we manage to end up with real NULLs in object registers 10:49
timotimo aye 10:53
i'll have a closerl ook
11:05 brrt left
Geth MoarVM: 0e94d75a7b | (Ben Davies)++ | build/setup.pm
Temporarily use -fno-ret-protector when building on OpenBSD

  See #1091		 11:11
MoarVM: e73e82fa36 | (Jonathan Worthington)++ (committed using GitHub Web editor) | build/setup.pm
Merge pull request #1092 from Kaiepi/retguard

Temporarily use -fno-ret-protector when building on OpenBSD
MoarVM: 09e1575355 | (Ben Davies)++ (committed using GitHub Web editor) | src/core/alloc.h
Remove unnecessary (void *) cast in MVM_free_zero
 11:12
MoarVM: d111dd9014 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/core/alloc.h
Merge pull request #1088 from Kaiepi/patch-1

Remove unnecessary (void *) cast in MVM_free_null
MoarVM/master: 6 commits pushed by (Vadim Belman)++, (Jonathan Worthington)++ 11:13
patrickb jnthn++ # Great you're working on getting the PR backlog smaller! 11:14
Geth MoarVM: cc06ad280e | (Patrick Böker)++ | Configure.pl
Never build a relocatable moar on OpenBSD

  See github.com/rakudo/rakudo/issues/2818
OpenBSDs implementation of $ORIGIN in rpath is unreliable. Also it misses a mechanism to determine the executable path in general. So even at runtime one has a hard time implementing something like this in a robust manner.		 11:16
MoarVM: 12e7590cf9 | (Jonathan Worthington)++ (committed using GitHub Web editor) | Configure.pl
Merge pull request #1089 from patzim/openbsd-no-relocatable

Never build a relocatable moar on OpenBSD
11:18 brrt joined 11:23 domidumont joined 11:37 zakharyas left
timotimo i don't really get it ... 11:43
the frame just sets up a callsite with one positional arg and a few nameds, passes a single object argument, and invokes what the frame got called with (after a decont)
the frame is <main>, the object it passes is what it gets from MVM_clargs 11:44
11:45 travis-ci joined
travis-ci MoarVM build passed. Jonathan Worthington 'Merge pull request #1089 from patzim/openbsd-no-relocatable 11:45
travis-ci.org/MoarVM/MoarVM/builds/521669706 github.com/MoarVM/MoarVM/compare/0...e7590cf9c1
11:45 travis-ci left
timotimo perhaps i'm looking at the wrong frame here? 11:48
11:48 ggoebel left
timotimo oh, wait, the callsite has a flag_count of 6, arg_count of 11, and num_pos of 1, but there's only a single argument being passed 11:54
so the reason why we get this problem is that the callsite used in the bytecode has a different amount of arguments than the bytecode 11:56
so the validator needs to be a bit more agressive when it comes to argument and invoke sequences 12:01
12:04 brrt left 12:06 brrt joined 12:20 brrt left 12:21 brrt joined
timotimo so it turns out the necessary code to make sure all flags actually have values passed is already in the validator, it just doesn't get called in the necessary place 12:58
spec tests don't seem unhappy yet (though i haven't run them all yet) 13:03
13:16 domidumont1 joined
timotimo i should perhaps have recompiled moar with C optimization before running a spec test, i now realize 13:19
13:20 domidumont left
timotimo that's a full pass! 13:20
Geth MoarVM: 654417932b | (Timo Paulssen)++ | src/core/validation.c
fuzzing: activate 2nd argument count check for invocations

until now the bytecode validator only made sure the number of positionals in the callsite matched the number of arg commands that corresponded to positionals, but there could be nameds that didn't have a corresponding arg command that would then leave a low-level null in object registers, which spesh would stumble over		 13:27
MoarVM: 8fef199947 | (Timo Paulssen)++ | src/6model/reprs/P6opaque.c
fuzzing: reject out-of-bounds unbox slot numbers
timotimo oh my, the cur_arg variable in the validator's "validate the argument passing ops" just counts from 0 to n and makes an index comparison there 13:37
so it doesn't catch when an argument op has an out-of-bounds arg index
13:42 lizmat left 13:49 MasterDuke left 13:54 zakharyas joined 14:04 patrickb left 14:30 brrt left 14:31 domidumont joined 14:34 domidumont1 left 14:42 zakharyas left 14:43 zakharyas joined 14:48 lucasb joined
japhb timotimo: BTW, huge fan of this fuzzing work. Thank you! 15:05
15:09 domidumont left
timotimo japhb: sadly the effects will barely be felt IRL :P 15:15
japhb timotimo: IRL I work for a company that is literally constantly under attack from major governments. Fuzzing seems incredibly useful to me. Also, as jnthn points out, some of the situations the fuzzer is pointing out aren't just cases of a missing null check (where a null would be rare but reasonable), but indications of something more serious wrong in that a null shouldn't even be possible at that point. 15:17
So it's catching real implementation issues, not just forgetfulness. 15:18
timotimo right, though the current fuzzing work mostly requires you to actually manipulate bytecode or the serialized blob :)
some things like nqp::elems(nqp::array) null-pointering can be reached
but not something like "the argument passing ops aren't proper" 15:19
japhb Also the company I work for deals with enough data for bit errors to be an effectively constant problem. We do a lot of ECC. ;-)
ugexe we are missing the most important part of fuzzing: the $some-amount of buzz it generates when someone eventually posts it on hackernews 15:22
timotimo hah 15:26
we don't have something for "the bytecode was correct when the validator saw it, but then it got corrupted in memory by a bit-flip" 15:27
ugexe: would be very nice to at some point fuzz actual perl6 code, but our run time speed isn't there yet 15:28
and of course the first run through the test corpus i'm using won't be terribly interesting from the standpoint of tickling deep bugs 15:30
since it's currently only doing one derivation step from what i put in (which is the stage2 of nqp) 15:31
it should also do some "mixing" near the end of the first run; cut portions out, move big chunks around, cross-over with other versions perhaps?
but since it's usually doing about 75 executions per second and i'm only utilizing a single of hack's 4 cores ... 15:32
ugexe is your local environment not sufficient? 15:33
timotimo it tends to be ram-starved because i use a web browser 15:34
tadzik ouch :D 15:35
timotimo 16 gig just ain't cutting it these days
tadzik huh
8 is usually sufficient for me
but I'm a person who closes tabs
timotimo i close tabs also, but i still have too many of 'em 15:37
tadzik :(
ugexe browse on hack via VNC and develop locally 15:38
japhb I sleep tabs, on machines where I have the proper extensions installed. That way I can get them back without too much trouble, thus keeping me from tab hoarding *too* strongly.
timotimo is gurgel chrome capable of that? 15:39
japhb timotimo: I use Tabs Outliner (free version) for Chrome. Some other people on my team use an extension that auto-sleeps any tab you haven't looked at in a while. Advantage of Tabs Outliner is that the tabs/windows you sleep actually disappear (becoming grayed out in the outline), it tracks what tabs launch other tabs, you can drag and drop tab hierarchies around easily, and you can sleep large swaths of 15:42
the tab outline all at once.
Advantage of their extension: You don't have to think about it at all, it's like having automatic memory and CPU reduction for Chrome. 15:43
Which, let's face it, is not terribly light on either.
timotimo ah, i also have tabs outliner
i barely open it
it's got about a hundred "crashed windows" that i never got around to cleaning up 15:44
japhb Heh
Would be nice if Chrome didn't produce quite so many of those ....
timotimo i don't recall a time it crashed on me because of chrome 15:45
usually it's one thread exhausting ram or something, then the core dump exhausts my hard drive, then chrome hangs and gets killed ... or dies? dunno.
ugexe hmmmm, what the hell sites are you vising that do all that... 15:47
timotimo oh, the sites don't do that 15:48
rakudo does
when i let it write extraordinarily big profiles or heap snapshots into /tmp, which is in my ram, for example
dogbert17 japhb: wrt R#2848 running your example with MVM_SPESH_INLINE_DISABLE=1 seems to resolve the problem 16:00
synopsebot R#2848 [open]: github.com/rakudo/rakudo/issues/2848 Writing to many synchronously run child processes grinds to a halt
japhb dogbert17: Oooh, that is very interesting. 16:15
dogbert17: Yup, confirmed even with 4 threads max. In fact, running with RAKUDO_SCHEDULER_DEBUG=1 RAKUDO_MAX_THREADS=4 MVM_SPESH_INLINE_DISABLE=1 shows no new threads at all. 16:18
16:26 lizmat joined 16:31 zakharyas left 17:15 lizmat left 17:31 brrt joined 17:48 lizmat joined 18:12 brrt left 19:05 lizmat left 19:15 zakharyas joined 19:23 zakharyas left 19:38 lucasb left 19:44 lizmat joined 19:52 brrt joined 19:54 lizmat left
brrt \o 20:09
20:16 AlexDaniel left 20:24 squashable6 left 20:27 squashable6 joined 21:00 brrt left 22:12 lizmat joined
samcv jnthn, how do you feel about having a thing to check for malloc_trim in configure.pm? 22:43
or whichever file it was :). since there seems to be no better way to detect this, but it would be nice to be able to ensure perl 6 lowers its memory usage. or at least what shows up 22:44
jnthn samcv: Yeah, a probe for its availability is fine. I think the only outstanding questions are how often to call it, and what the value to it should be. I think it's pretty much agreed we want it. 22:57
Also, maybe the best way to determine those two will be to get it in and prof real programs
samcv yeah 22:58
jnthn, sounds good. will look into that 22:59
jnthn samcv++
samcv maybe we want to make sure not to call it more than X number of ms or something 23:00
jnthn Putting it after GC is probably reasonable, though question is if it's every run, every N runs, or every full (non-nursery) collection 23:01
japhb jnthn: While every full collection seems like a good default, I worry about further extending the difference in pause time between nursery and full collections. Animation code I've written that shows frame-by-frame delay shows a clear signal of pauses every ~N frames long enough to be perceived as visible stutter; my best guess (until we have telemetry of some sort for when GC pauses happen and what type) 23:13
is that these are the full GC cycles. 23:14
Oddly, for animations it's often OK to have lower but steadier frame rate.
(As a side note, a pause of just 16-17 milliseconds represents a full dropped frame at 60 fps, so it doesn't take a very long pause to make a very big stutter.) 23:16
jnthn japhb: Hmm, does such code normally have a lot of objects that would live long enough to get promoted? 23:18
Perhaps, given Perl 6 being allocation heavy, the answer is "yes, until EA is doing a better job" :) 23:19
I ask because full collects are not just "every N nursery collects", but based on the rate of object promotion
A program that never promotes anything will potentially never do a full collect
That's also an interesting point were we to pick doing it every gen2: what of programs that manage to fragment unmanaged memory while never causing enough gen2 promotion to trigger the malloc_trim? 23:21
japhb jnthn: Yeah, that latter bit is true. :-) But actually, I wonder if there isn't some amount of "There's enough in Gen 2 after a while that full collections are just slow even if the fraction of new objects is low." (I get your point about a certain number of object promotions being necessary to trigger full GC, and I'm not sure what exactly is getting promoted each time.)
(latter bit == promotions common until EA does better)
Oh yeah, fragmented unmanaged memory is an interesting question. 23:22
jnthn The EA work is looking quite promising, it's just tricky :)
japhb I BET!
jnthn: Are you holding back merging EA until it is "more correct", "more effective", or "after the release cycle"? 23:24
jnthn japhb: I'm working on it in phases, merging each phase "when it's ready". The initial one was "do some EA in very simple cases"; that's already in master. 23:29
japhb jnthn: Yeah, that one I'd seen. I guess I meant the branch you've been working on a while. 23:30
jnthn The current phase is: analyzing transitive references, P6bigint handling, and making it handle partial escapes, which together should be effective in a lot more cases.
Transitive references: the analysis is correct, but deopt needs adaptation. 23:31
P6bigint handling: just got some first cases of that working. Will probably handle a few more, though doing them all isn't a pre-req for merge. BUT it does need further work to make sure it won't ever leak the unmanaged mp_int buffers. That's quite important to get right. :)
japhb oh yeah, definitely. 23:32
jnthn Partial escapes: actually the algorithm we're using was based on one found in a paper on partial escape analysis, so a lot is set up for this already. It'll also make things vastly more useful.
japhb nodnod 23:33
jnthn Example of the latter: array indexing. We inline the fast-path common case. But the Int index can escape to the slow path (out of bounds erorr reporting, etc.)
Partial escape handling lets us defer the allocation to the point we know it's escapging 23:34
*escaping
But also, if we have a bunch of integer calculations and store the final result, we also want to materialize the final result at that point too; at the moment we can't do that, so end up with the whole chain of operations being considered escaping. 23:35
Once I get those 3 in decent shape, I'll merge. I hope to make it in time for the 2019.5 release, but won't be rushing.
The phase after that will be about handling EA in code involving loops and OSR. 23:36
Once that is done, we have a fairly complete overall structure, so it'll be a case of smaller situation-specific additions. 23:37
For example, extending it to cover things other than P6opaque, such as hashes indexed entirely with constant keys 23:38
The hash case is very common in object construction; in principle, the slurpy hash and the indexing of it could in some cases entirely go away. 23:39
japhb Oh nice, that would probably have quite a big effect. 23:53
jnthn Hope so. :) 23:58
sleep o/

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!