Welcome to the main channel on the development of MoarVM, a virtual machine for NQP and Rakudo (moarvm.org). This channel is being logged for historical purposes.
Set by lizmat on 24 May 2021.
00:02 reportable6 left 00:05 reportable6 joined 00:35 evalable6 joined 01:36 linkable6 joined 04:38 evalable6 left, linkable6 left 04:40 evalable6 joined
Nicholas #moarvm *, good 05:25
05:40 evalable6 left 06:02 reportable6 left 06:04 reportable6 joined 06:41 Altai-man left, linkable6 joined 06:42 sena_kun joined 06:43 evalable6 joined 07:13 moon-child left, moon-child joined
Nicholas was it something I said? :-) 07:21
07:23 MasterDuke left 07:28 MasterDuke joined
nine good, good, #moarvm 07:43
MasterDuke #moarvm good, coffee good, t/02-rakudo/15-gh_1202.t segfaults bad 07:46
Nicholas so that bug was rather hard to figure out. In future, can we stick to creating easy bugs please? Thanks in advance :-) 07:47
*I* thought that jnthnwrthngtn had some idea about causes of (at least one class of) bugs in t/02-rakudo/15-gh_1202.t and that something big blocked fixing cleanly. But I forget whether that was new-disp, Raku-AST, $other, or "I'm confused and it wasn't" 07:48
nine MasterDuke: is there something special required to reproduce that segfault?
MasterDuke you'd have to ask dogbert17. i tried running a stripped down t/02-rakudo/15-gh_1202.t, but never got a segv. maybe i'd taken too much out 07:49
but i've seen tens of times now in the CI runs for my prs 07:50
nine 15-gh_1202.t has always been troublesome on s390
MasterDuke timo was chatting with dogbert17 about it recently, don't remember if you were in that conversation
colabti.org/irclogger/irclogger_lo...10-03#l112 07:51
i very rarely hit it locally 07:52
08:08 MasterDuke left 08:12 TempIRCLogger__ left, Nicholas left 08:21 MasterDuke joined
MasterDuke i've also seen a couple random fails due to t/02-rakudo/reproducible-builds.t 08:21
08:24 TempIRCLogger__ joined, Nicholas joined 08:48 ggoebel left
jnthnwrthngtn moarning o/ 09:24
tellable6 2021-10-04T18:10:00Z #moarvm <vrurg_> jnthnwrthngtn Is there any reason to have a unique per-compunit GLOBAL at compile time and only install the primary one for run-time? I think we have an error caused by this design.
Nicholas \o 09:25
jnthnwrthngtn: it compiles [the setting] - ship it!
jnthnwrthngtn Goodness folks typed a lot here while I was having a long weekend away...
Nicholas a fun trip? 09:26
jnthnwrthngtn .tell vrurg It's about separate compilation; the compilation of a module should start with an empty GLOBAL and accumulate the things that are `use`d by it. 09:27
tellable6 jnthnwrthngtn, I'll pass your message to vrurg
nine jnthnwrthngtn: sorry, didn't know we should take a break :)
jnthnwrthngtn Nicholas: "it compiles" - on little endian? 09:28
Nicholas: Yes, there was good food and good beer and some nice places to walk around. :) 09:29
Nicholas both!
jnthnwrthngtn Oh, wait, big endian is the one it was busted on... :)
I guess this coffee isn't working yet.
Nicholas (we don't talk about mixed endian)(we don't need to. UV doesn't do PDP-whatever-it-was)
jnthnwrthngtn Gah, you beat me to "only both" :P 09:30
Nicholas actually, also, I have no idea if anyone has produced a C compiler for the crazy legacy things that defines an ordering in memory for 64 bit integers
nine jnthnwrthngtn: Is this a sensible approach? github.com/rakudo/rakudo/compare/n...nativecall Now working on that foreign-func dispatch terminal 09:32
jnthnwrthngtn Looking at github.com/MoarVM/MoarVM/pull/1557, 1) sorry, I can't count, 2) maybe we should get update_ops.raku to spit out constants for op length and use them in the interpreter.
MasterDuke something like `cur_op += MVM_OP_LENGTH_decont`? 09:34
jnthnwrthngtn nine: Hmm. That CUSTOM-DISPATCHER thing probably works although I note that means you can never .wrap a native routine
MasterDuke: Something like
nine: Though maybe that's "no, we don't support that" :) 09:35
nine jnthnwrthngtn: it does work. But yes, that wrapping issue is a drawback
jnthnwrthngtn Well, I mean "works as a design", it looked like it'd work as in actually function
nine Ok. Well, it's my first idea for a hook for custom dispatchers 09:36
jnthnwrthngtn An alternative idea I had was a registry of role + dispatcher name, and if there are entries in that registry, before going down the raku-invoke path for a Routine, we'd see if one of those roles is done, and if so use the alternative dispatcher.
(That is, we'd end up in raku-invoke, and this would be a final thing before we do the default thing of "extract $!do and invoke it" 09:37
)
That'd mean we're already past the wrapping, multis, etc.
Nicholas jnthnwrthngtn: I wasn't sure if it was worth updating all the things to also generate constants for op size and then churning all the C code. 09:38
jnthnwrthngtn Yeah, it's quite the churn
nine I could also shuffle around the code a bit to achieve the same with CUSTOM-DISPATCHER. I used that for wrap because it nicely demonstrates the capability and to make it performance neutral with what we had
Nicholas it takes quite a lot of unfortunate circumstances simultaneously to manage to get it wrong, but not have immediate catastrophic failure 09:39
dogbert17 jnthnwrthngtn: github.com/rakudo/rakudo/issues/4548 might be of interest after your coffee has started to take effect
jnthnwrthngtn Nicholas: Fair.
nine If we wanted wrappable native routines, all I have to do is bring the special casing of WRAPPERS back to the dispatchers
jnthnwrthngtn nine: I think this extension point should be looked for in raku-invoke, not in raku-call and several other places 09:40
nine Should .wrap still use the extension point or should I just revert that part? 09:41
jnthnwrthngtn It can be a method, I just kinda wanted to keep stuff specific to dispatchers out of userspace
I'd keep it distinct from .wrap handling
nine k
jnthnwrthngtn We've already had situations in the past where folks grumbled that, for example, multi dispatch didn't properly compose with native calls 09:42
Probably we shouldn't create the same limitation on wrappers
(I think the multi case is all fixed up now, though)
MasterDuke re churn, --ignore-rev (or the file of commits to ignore) introduced in git 2.23 could be useful 09:43
jnthnwrthngtn Nicholas: So, the endian stuff is fixed enough I probably don't need to follow it too closely when reading up on things, or there's still outstanding issues there? 09:45
Nicholas it's fixed.
you can enjoy the chaos, without needing to think to hard about the details 09:46
too hard
09:48 linkable6 left, evalable6 left
MasterDuke interesting, github.com/MoarVM/MoarVM/pull/1555...85ec768341 reliably succeeds on linux/mac, but reliably dies in stage parse on windows with `NMAKE : fatal error U1077: 'C:\Strawberry\perl\bin\perl.exe' : return code '0xff'` 10:09
i hoped github.com/MoarVM/MoarVM/pull/1555...09dda2651d would fix it, but nope 10:10
jnthnwrthngtn github.com/rakudo/rakudo/issues/4549 is rather odd 10:15
"failed to load library 'dynext\perl6_ops_moar.dll'"
MasterDuke and it doesn't happen in our CI jobs 10:16
nine Yeah, I suspect a local issue 10:17
dogbert17 nine: I haven't found a good way to repro the SEGV, the best I have is: 'while MVM_SPESH_NODELAY=1 ./rakudo-m -Ilib t/02-rakudo/15-gh_1202.t; do :; done'
nine dogbert17: k
dogbert17 it tends to fail after 10-20 iterations
worse is the fact that running this test under rakudo-gdb-m leads to a hang on my machine (deadlock ?) 10:19
nine jnthnwrthngtn: putting the CUSTOM-DISPATCHER check in raku-invoke has a bit of a downside as well: one can then not just delegate to raku-invoke to actually invoke the original body 10:24
Of course it's not impossible. But a bit inconvenient and takes a little more dibbing into internals 10:25
I guess that's worth being able to wrap native routines though. I can easily see a use for that. 10:26
jnthnwrthngtn nine: Hm, would we want to invoke the original body? Not in the NativeCall case at least... 10:28
nine: But if you have the $!do then it's a case of feeding it to boot-code, so maybe not a huge pain.
nine Certainly not for NativeCall. But since it's a generic hook, I can see cases where one would just mangle the arguments a bit. Or only redirect dispatch in some cases but not in all of them. 10:29
But yes, it's an inconvenience but not a blocker.
jnthnwrthngtn We could in principle insert yet another layer, but while that'd have no impact once the dispatch program is recorded, it would have one on callsite setup time, which is worth some consideration. 10:32
(Since we've already regressed startup time an amount) 10:33
dogbert17: The coffee apparently sufficed for #4548. :)
nine That's why I figured using the generic hook for wrappers would be a good idea, since the number of checks we do for an unwrapped case would have been exactly the same 10:34
But then I'll leave it up to MoarVM hackers to make it fast anyway :D 10:35
Wait.....that'd be us again
MasterDuke jnthnwrthngtn: fyi, new-disp broke the rakudo jvm build. part of it was figured out (removed some nqp::getcoderef's to follow a change to the moarvm specific code), but it currently dies trying to do some role mixin
bartolin said he'd try and fix it 10:36
jnthnwrthngtn MasterDuke: Argh, so I busted that too. :( Sorry. 10:37
dogbert17++ for digging up lots of older issues that are fixed by new-disp 10:41
MasterDuke no worries (and further fyi, the part that was figured out hasn't been committed, because i wasn't 100% sure it was correct)
nine jnthnwrthngtn: pushed an updated commit implementing what we discussed 10:44
jnthnwrthngtn Thanks, I'll take a look after lunch 11:01
dogbert17 jnthnwrthngtn++: coffee for the win :) 11:06
nine: did you see gist.github.com/dogbert17/a07d0bb2...66c9d6dc8c 11:07
nine dogbert17: seen it. Trying to reproduce. But I'm tempted to leave it for when I'm back home. The factor 8 difference in processing power between my laptop and desktop does make a noticable difference in fun investigating such issues 11:22
MasterDuke nine: you still have a bunch of tabs in that commit, down to ~line 680 11:33
nine Oh, indeed. Must have turned off paste mode too late or something 11:40
Pushed yet again 11:41
jnthnwrthngtn: your new NativeCall tests seem to fail on s390x: build.opensuse.org/public/build/ho...akudo/_log 11:44
Native callback must be a code handle 11:45
11:58 Xliff joined
Xliff Hello, #moarvm 11:58
12:02 reportable6 left 12:03 reportable6 joined
jnthnwrthngtn nine: huh, that's odd 12:03
nine: On that platform is it using libffi rather than dyncall? 12:06
Geth MoarVM: e28e06ded2 | (Jonathan Worthington)++ | src/core/nativecall_libffi.c
Correct libffi/dyncall discrepancy
 12:08
jnthnwrthngtn Hopefully that will do it
MasterDuke nine: the build log seems to show it's using an old nqp and moar for the build
[ 79s] ===/usr/bin/nqp-m version 2021.09 is outdated, 2021.09-125-g6add04b expected. 12:10
[ 79s] ===
[ 79s] Using /usr/bin/nqp-m (version 2021.09 / MoarVM 2021.09).
nine MasterDuke: I noticed. But if it were that old, everything would fail because of missing nqp::dispatch 12:12
So I figure that's a bogus message
MasterDuke right 12:13
nine I think I have all the code together for a boot-foreign-code dispatch terminal which ends up calling MVM_nativecall_invoke. Now to wire it up at the higher level and see how it explodes 12:21
jnthnwrthngtn nine++ 12:26
nine: At github.com/rakudo/rakudo/compare/n...6R271-R275 it's assumed that the thing being invoked is a always constant at that callsite, but that may not actually be the case; in principle one could do `my &callee = $foo ?? &one-native-sub !! &another-native-sub; callee(42)`. 12:35
nine jnthnwrthngtn: but callee is already tracked? If we end up calling a different native sub, shouldn't that guard trigger? 12:42
Xliff dir 12:43
12:49 Xliff left 12:50 evalable6 joined 12:51 linkable6 joined
ugexe . .. illegal_files Documents 12:54
jnthnwrthngtn nine: I believe its type is tracked, but I don't see any literal guard being established. 13:03
13:05 ggoebel joined
jnthnwrthngtn See at github.com/rakudo/rakudo/blob/mast....nqp#L2467 for how we branch over the two situations as an optimization 13:05
(Thus telling the runtime that if the Code object is a constant then we can assume the $!do is a constant, but otherwise we can't assume that and have to track and use the $!do) 13:06
For the native call case, we can in the constant case further assume that the native call is already set up, but should probably guard against that in the non-constant case 13:07
(By checking some attribute that will be unset if it's not setup for example)
nine ok
[Coke] anyone see anything wrong with this dumb batch file? gist.github.com/coke/d3ef45869600e...bd9325ac0d 13:10
Trying to do separate builds into a specific dir, but the last rakudo step there seems to be ignoring the INSTALL_DIR\bin in the path.. 13:11
13:11 patrickb joined
[Coke] with the prefix specified, I am not sure the PATH even matters. 13:13
MasterDuke yeah, i don't think the you need the prefix in the path
i don't have the prefix i use in the path (on linux) 13:14
on a different topic, my jit of scsetobj, scsetcode, and getcomp all reliably fail on windows 13:15
so i think i'm going to remove them from the latest PR so it's clean, and then hopefully it can be merged and the others figured out in a separate PR 13:16
[Coke] jnthnwrthngtn: ah. I think I just cut and pasted from the wrong build failure window. 13:25
I'll try to update with the correct trace and the data on the binaries you asked for
13:41 ggoebel left 13:54 sena_kun left 13:55 sena_kun joined 14:03 Altai-man joined 14:04 sena_kun left
MasterDuke ok, github.com/MoarVM/MoarVM/pull/1555 is green across the board. anybody care to take a look? 14:09
14:15 Altai-man_ joined 14:17 Altai-man_ left
Geth MoarVM: MasterDuke17++ created pull request #1558:
Change CI job strategy from matrix to parallel		 14:28
14:52 ggoebel joined 14:57 Xliff joined 15:11 ggoebel left 15:13 ggoebel joined 15:46 patrickb left
nine So the boot-foreign-code dispatcher seems to work and calls MVM_native_invoke and the native function runs and gets correct parameters. But then it fails with: "Dispatch callback failed to delegate to a dispatcher" 16:23
jnthnwrthngtn nine: Did you do all of 1) add a new kind of dispatch program outcome, 2) add a function to set that, and have your boot-foreign-code implementation call it, 3) add a new dispatch op, 4) wire up the dispatch program assembler to spit that op out? 16:29
(The outcomes are 1:1 with the dispatch terminals, pretty much) 16:30
nine jnthnwrthngtn: I may be missing step 4. 16:32
jnthnwrthngtn Ah, there's also a 2.5, which is to handle it in the switch statement that processes the different outcomes in the record phase. 16:33
Which is probably the one with a `default` that spits out the error you see 16:34
nine The error I get is from MVM_disp_program_record_end case MVM_DISP_OUTCOME_EXPECT_DELEGATE: 16:35
jnthnwrthngtn Oh. That is not the outcome a terminal should have
Can I see a diff or something?
nine gist.github.com/niner/ce782bfcf5ee...be78e34375 16:36
The actual call happens in case MVM_DISP_OUTCOME_FOREIGNFUNCTION: 16:37
jnthnwrthngtn Ah, right, we don't actually invoke bytecode 16:45
ah, but the unwind is being handled 16:46
This certainly doesn't happen on some follow-up call, e.g. control never returns after the native call? 16:48
Hm, I don't spot the problem. What's on the C callstack when we oops? 16:51
nine It does reach the "note "dispatch ended";" after the "nqp::dispatch('boot-foreign-code', $callee.call, $new_capture)" in the raku-nativecall dispatcher on the first run. But does not succeed in printing the result of the native call. 16:53
Backtrace: gist.github.com/niner/c281cb03ce3a...c919ba79bf 16:54
jnthnwrthngtn Sure, but dispatcher-delegate doesn't actually call a dispatcher, it just sets where we will delegate 16:55
Ah, I have a guess; what does the Rakudo side of this look like?
You'd need to delegate to boot-foreign-code, not use it 16:56
That's probably the issue
nine gist.github.com/niner/89b2a8996f0c...5a59ad0e28
jnthnwrthngtn ah yes 16:57
nqp::dispatch('boot-foreign-code', $callee.call, $new_capture);
Should be:
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-foreign-code', $delegate_capture);
nine Aaah...that even makes sense
jnthnwrthngtn Where $delegate_capture is $new_capture, but with $callee.call inserted at the start
nine Turns out I had the correct expectation when I wrote the original code for MVM_nativecall_args. But then I adjusted the code to the faulty caller :) 17:06
Oh, very nice. 17:21
Tests start passing :) What's missing now is deconting of arguments 17:22
jnthnwrthngtn yay :) 17:26
nine: For Scalar we can just do it as part of the dispatch program. For Proxy, probably it needs something like the multi-dispatcher trick :)
(With the ProxyReaderFactory and the resumption)
nine I'm stealing pass-decontainerized for now 17:28
jnthnwrthngtn Yeah, that'll do it :) 17:30
17:38 linkable6 left, evalable6 left
nine Almost all tests passing now :) 17:41
jnthnwrthngtn Wow. nine++
Now "all" that's left will be to add a sp_nativecall or so, translate it, and JIT it :D 17:42
nine Need to port that callback logic as well
Child's play :D
jnthnwrthngtn Ah yes, to unwrap the code object
Also vastly cheaper :)
(To do it in the dispatch program)
nine Alas, despite the Deutsche Bahn doing its utmost to give me more time for hacking, even this train ride ends soon. But then, I got some 2.5 extra hours, so I shouldn't complain ;) 17:43
Nicholas will they also pay you for the extra hacking time they gave you? 17:44
nine If I manage to get past their bureaucracy. Raku - sponsored by Deutsche Bahn :D 17:46
MasterDuke has all the low-hanging fruit in been picked? 17:51
*in MVM_string_utf8_decodestream
huh. perf thinks 42% of its time is spent on `mov 0x30(%rsp),%rdi` 17:58
18:03 reportable6 left
MasterDuke huh, i think it's saying that's in the slow path, but the slow path was only taken 60k times (compared to 45m for the fast path) 18:04
18:05 reportable6 joined 18:10 [Coke] left 18:13 [Coke] joined 18:33 tealecloud joined 18:39 ggoebel left, ggoebel joined, linkable6 joined 18:41 evalable6 joined
lizmat SPA: the Rakudo Weekly News is like the Deutsche Bahn: delayed (until tomorrow) 18:48
MasterDuke we may want to investigate switching utf8 decoders. the benchmark at github.com/bdonlan/branchless-utf8 gives me 541.333398 MB/s for the one we use, but 708.000084 MB/s for a branchless version and 945.333446 MB/s for a simd version 18:50
slightly different numbers, but same ordering with both gcc and clang 18:52
lizmat don't we have a custom decoder that handles NFG ? 18:54
MasterDuke github.com/MoarVM/MoarVM/blob/mast...8.c#L3-L63 is what we use, it's tested in that benchmark 18:56
[Coke] ugh. I now have no working raku installed on my windows box. 18:57
(managed to eff up my "system" install while testing out HEAD)
18:59 ggoebel left
MasterDuke whoops, that fork of the benchmark that added the simd version had an older version of the one we're using. with the correct version of what we're using, i get 697.333416 MB/s. that's slightly faster than the branchless version, so no reason to switch to that. but the simd version is ~250MB/s faster 19:19
19:19 moon-child left, moon-child joined 19:26 childlikempress joined, moon-child left 19:43 childlikempress left 19:47 moon-child joined
MasterDuke dirtyhandscoding.github.io/posts/u...ction.html looks like it might be another good option 19:55
20:02 moon-child left 20:03 moon-child joined
jnthnwrthngtn I suspect the performance challenge with UTF-8 decoding in MoarVM is not so much the UTF-8 decoding into codepoints, but rather the NFG calculation from those codepoints. 20:06
MasterDuke of course the actual utf8 decoding isn't really our bottleneck. using the benchmark program from ^^^ i read the 1.5gb spesh log file i've been working with in 4.2s using the decoder we use, and 3.2s using one of the faster ones 20:07
but `Nil for "spesh.log".IO.lines; say now - INIT now` takes 32s
oh this is interesting 20:13
`time ./rakudo-m -ne 'Nil' spesh.log` takes 23s 20:14
`time ./rakudo-m -e 'Nil for "spesh.log".IO.lines;'`
takes 33s
20:15 ggoebel joined
[Coke] ok, I tried to do a clean build of rakudo with 2021.09 rakudo (which pulls in the same tag from nqp/MoarVM. I *still* get the failure about perl6-ops. wtf. 20:16
20:29 moon-child left, childlikempress joined
timo double-checked the install paths? is there some kind of debugging tool like LD_DEBUG=all or whatever for windows dll loading? 20:54
to see where it's looking and not finding it?
actually, i don't even know if finding the file is the issue 20:55
[Coke] failed to load library 'dynext\perl6_ops_moar.dll' 20:57
that file exists and appears to be of the right type.
I have tried a fresh command prompt to avoid lingering env vars. 20:58
timo does the error come from dynload or whatever lib we're using to dynamically load libraries 20:59
21:03 ggoebel left
[Coke] Don't know. Trying 2021.08 to see if I can just get a working raku again, either way will see if I can get a debug build to tell me what's going on at HEAD 21:05
timo: I can do a moarvm build with --debug and then I can step through. 21:09
21:16 ggoebel joined, childlikempress is now known as moon-child 21:30 MasterDuke left 22:02 MasterDuke joined 23:02 tellable6 left, bisectable6 left, releasable6 left, notable6 left, unicodable6 left, sourceable6 left, nativecallable6 left, shareable6 left, committable6 left, bloatable6 left, squashable6 left, reportable6 left, statisfiable6 left, greppable6 left, linkable6 left, quotable6 left, benchable6 left, coverable6 left, evalable6 left 23:03 benchable6 joined 23:04 sourceable6 joined, bisectable6 joined 23:05 releasable6 joined, notable6 joined, bloatable6 joined 23:06 evalable6 joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!