Welcome to the main channel on the development of MoarVM, a virtual machine for NQP and Rakudo (moarvm.org). This channel is being logged for historical purposes.
Set by lizmat on 24 May 2021.
00:07 reportable6 left 00:10 reportable6 joined 00:35 kjp left
timo that sounds like a good idea actually, yeah 00:47
02:18 sourceable6 left, bloatable6 left, notable6 left, bisectable6 left, benchable6 left, unicodable6 left, greppable6 left, squashable6 left, linkable6 left, statisfiable6 left, quotable6 left, shareable6 left, tellable6 left, evalable6 left, coverable6 left, nativecallable6 left, releasable6 left, reportable6 left, committable6 left 02:19 bisectable6 joined, greppable6 joined, quotable6 joined, notable6 joined 02:20 unicodable6 joined, coverable6 joined, shareable6 joined, reportable6 joined 03:19 bloatable6 joined, evalable6 joined, committable6 joined 03:20 statisfiable6 joined 03:21 linkable6 joined, sourceable6 joined 04:08 kjp joined 04:20 tellable6 joined, nativecallable6 joined, releasable6 joined, benchable6 joined 05:22 evalable6 left, committable6 left, tellable6 left, nativecallable6 left, greppable6 left, benchable6 left, bisectable6 left, statisfiable6 left, releasable6 left, notable6 left, sourceable6 left, linkable6 left, reportable6 left, bloatable6 left, shareable6 left, quotable6 left, unicodable6 left, coverable6 left 05:23 coverable6 joined, unicodable6 joined, bisectable6 joined 05:24 committable6 joined, quotable6 joined, sourceable6 joined 05:25 benchable6 joined, evalable6 joined, linkable6 joined, releasable6 joined 06:24 tellable6 joined 06:25 notable6 joined, nativecallable6 joined 06:57 bartolin left 06:59 bartolin joined 07:21 squashable6 joined
Geth MoarVM: MasterDuke17++ created pull request #1633:
Simplify MVM_string_utf8_encode_C_string
08:23 statisfiable6 joined 08:24 greppable6 joined 08:25 bloatable6 joined
timo i hope we're not using that function in performance-critical code tho :) 08:34
MasterDuke it's used in a few more places than i expected, but i don't think any are terribly performance critical in "usual" code 08:36
samcv: you might find github.com/dotnet/runtime/pull/63285 and/or github.com/smart-tool/smart interesting 08:40
timo: while you're here, any thoughts about my get-and-reset-profiling-data-with-a-syscall experiment? is the naive MVM_profile_end+MVM_profile_start just not even going to work for experimentation and i really need to implement a correct solution? 08:42
timo i'm also stumped why the profile_end would return an empty object, though i didn't look at the actual code yet 08:43
nine Is realloc actually slower than malloc + memcpy?
timo realloc should sometimes be able to just re-use the memory and not memcpy 08:44
when the allocated piece was already big enough to hold one more byte anyway
MasterDuke i'd suggest that unless the realloc is in fact quite a bit slower (which seems unlikely), it's still a logical simplification of the code 08:48
nine: btw, have you seen that there have been a couple recent CI fails in t/02-rakudo/reproducible-builds.t ? 08:49
timo we don't implement realloc ourselves, right? only in like the fsa and such?
MasterDuke correct 08:50
nine: think it would make sense to add something like `MVM_dump_backtrace(tc)` to the hash iterators in moarvm, compile nqp and rakudo, and see where they're being called from without being sorted afterwards? 08:59
nine definitely 09:01
09:08 reportable6 joined
Geth MoarVM: 99e0a715d9 | (Daniel Green)++ | src/strings/utf8.c
Simplify MVM_string_utf8_encode_C_string

By just reallocing the result of MVM_string_utf8_encode and sticking a NULL on the end of it instead of mallocing a new larger string and memcpying into it.
MoarVM: 76051fcf3c | MasterDuke17++ (committed using GitHub Web editor) | src/strings/utf8.c
Merge pull request #1633 from MasterDuke17/simplify_MVM_string_utf8_encode_C_string
09:25 shareable6 joined
MasterDuke while we're discussing PRs, any comments on github.com/MoarVM/MoarVM/pull/1632 ? 09:29
Geth MoarVM: f4b4692ae4 | (Daniel Green)++ | .gitmodules
Ignore untracked changes in the ryu submodule

This stops the `modified: 3rdparty/ryu (untracked content)` message from showing up in a `git status` in the MoarVM repo after it's built. An alternative would be to add `ryu/ds2.o` to 3rdparty/ryu/.gitignore, but that would require adding a commit to the submodule instead.
MoarVM: 39e9a6be0c | MasterDuke17++ (committed using GitHub Web editor) | .gitmodules
Merge pull request #1632 from MasterDuke17/ignore_untracked_changes_in_ryu_submodule
MasterDuke hm. i have 470 distinct `at <...>` lines for the nqp build and 535 for rakudo (but the build was killed before stage parse finished). i'm not sure how to tell which ones to look into 09:51
e.g., could github.com/rakudo/rakudo/blob/mast...#L710-L712 be problematic? 09:55
nine No, it just binds those keys in another hash 09:56
MasterDuke hm, this is the only fail i could quickly find in the CI logs, but maybe it's not actually a reproducibility problem? dev.azure.com/Rakudo/rakudo/_build...amp;l=2058 10:14
nine 2022-01-03T22:07:44.5816545Z ===SORRY!=== Error while compiling D:\a\1\rakudo/t/02-rakudo/reproducible-builds.t 10:24
That doesn't look like it's due to an actual reproducible build issue
MasterDuke unrelated, but github.com/rakudo/rakudo/blob/mast...akumod#L51 is using the deprecated `.status`
nine Why is .status deprecated in the first place? 10:25
MasterDuke yeah. i'm going to put this investigation on hold until/unless we see another real reproducibility fail
from the deprecation message `Please use exitcode and/or signal methods (status is to be removed in 2022.06) instead.` 10:26
10:27 frost left
nine But exitcode and signal do different things than status 10:27
MasterDuke github.com/rakudo/rakudo/pull/4442 10:28
github.com/rakudo/rakudo/issues/1366 and github.com/Raku/doc/issues/1710 10:29
looks like a question for moritz
nine None of that explains why it has to go. Only that it's not wanted.
MasterDuke istr something about not all platforms setting the right values, so you need to inspect the two parts manually depending on what os you're on? 10:32
10:37 frost joined
MasterDuke some relevant chat here colabti.org/irclogger/irclogger_lo...07-27#l103 10:39
12:04 reportable6 left, linkable6 left 12:05 linkable6 joined 12:06 reportable6 joined
MasterDuke oh interesting. i fired up my mimalloc branch again and everything built ok. i think it's because previously i had changed the runners to use the MVM_* functions instead of e.g., malloc directly. however, that really shouldn't be required to get the benefit of using mimalloc, so i'll probably just leave that for a potential future change 12:22
i guess the thing to do now is decide if i/we only want to use mimalloc if a probe in Configure for cmake succeeds 12:24
however, i just recently noticed github.com/microsoft/mimalloc#single-source which may mean we wouldn't need to depend upon cmake at all 12:25
12:41 linkable6 left
nine MasterDuke: oh, that sounds way better :) I guess mimalloc would then be just another part of libmoar.so 13:14
MasterDuke i assume that would increase the size of libmoar.so by the same amount as libmimalloc.a (234k), but the tradeoff seems worth it 13:16
now trying to figure out how to reconfigure the build to just build that single file 13:17
moon-child I'm pretty sure that's smaller than the unicode tables :P
MasterDuke heh 13:18
nine MasterDuke: I think all it needs is -I3rdparty/mimalloc/include -I3rdparty/mimalloc/src in CFLAGS and 3rdparty/mimalloc/src/static.c in the sources list for libmoar 13:20
MasterDuke i was wondering about that. it looks like for some of the other single file 3rdparty libs (e.g., tinymt) we do create .a's 13:21
oh, everything seems to be fine now 13:39
a downside however...mimalloc appears to not (yet) play nicely with valgrind 13:40
13:40 frost left
MasterDuke github.com/microsoft/mimalloc/issues/251 13:41
13:43 linkable6 joined 13:44 linkable6 left 13:45 linkable6 joined 13:46 Geth left, Geth joined 13:57 Geth left, Geth joined
MasterDuke hm. but now i get segfaults with --full-cleanup 14:00
14:01 Geth left, Geth joined 14:05 Geth left, Geth joined 14:10 Geth left, Geth joined
nine Can we disable mimalloc in valgrind builds? 14:13
MasterDuke well, it looks like someone is working on integrating mimalloc with valgrind's api
but yeah, i guess that could be a workaround for now 14:14
14:14 Geth left, Geth joined
MasterDuke but first i need to figure out why i'm getting these segfaults now 14:14
happening in MVM_gc_global_destruction 14:17
so far always seems to be here github.com/MoarVM/MoarVM/blob/mast...odel.c#L60 14:25
and the st looks ok. debug_name = 0x555555559c60 "BOOTStrArray" 14:26
but there's a warning printed before the segv: `mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x555555559c60`
hm. the other fields do have addresses that look more like this though `boolification_spec = 0x200004e0760` 14:29
looks like st->debug_name can be set by MVM_serialization_read_cstr, MVM_string_utf8_encode_C_string, and strdup 14:31
wonder if it's the strdup causing the problem, since i'm not using mimalloc to override the usual functions 14:32
ah-ha! they provide a `mi_strdup` 14:34
and now spectests are good 14:41
nine Does it change setting compilation times? 14:42
MasterDuke yes. doesn't seem to speed up a spectest as much now, but that might because only libmoar is using it, not everything when i LD_PRELOADed it 14:47
there have been some changes to moarvm and rakudo since i ran the numbers in github.com/MoarVM/MoarVM/issues/1623, but it looks like it's still about 1-1.5s faster at stage parse 14:49
without the change to convert the fsa to just forward on to the regular functions
nine: btw, do you happen to have any benchmarking code i could try like jnthnwrthngtn mentioned in his comment on that issue? 14:54
just tried with the sort-of-disable-fsa patch, times seem pretty much the same (i.e., mimalloc w/ normal fsa is same as mimalloc w/ disabled fsa)
so with this we could even rip out the fsa completely 14:55
nine MasterDuke: sorry, benchmark availability has long been a sore point for Rakudo 15:17
MasterDuke yeah. another long-standing project i contemplate occasionally is to get that daily moarvm bench back up 15:19
Geth MoarVM: MasterDuke17++ created pull request #1634:
Add mimalloc submodule
MasterDuke huh, `3rdparty\mimalloc\src\static.o:static.c:(.text+0x451d): undefined reference to `BCryptGenRandom'` 16:03
afk for a while, will have to see about that this evening 16:06
lizmat ok, this is weird: trying to get IO::Path!SET-SELF below inlining limit 16:18
with this signature: method !SET-SELF(str $path, IO::Spec $SPEC, Str:D $CWD)
the size is 340 bytes
with this signature: method !SET-SELF(str $path, $SPEC, $CWD) 16:19
the size is 446 bytes
how can the bytesize **increase** by removing constraints in the signature ? 16:20
nine That can only be answered by looking at the actual generated speshed bytecode 16:36
16:54 linkable6 left, linkable6 joined 16:57 Geth left, Geth joined
jnthnwrthngtn lizmat: When the type is not iterable, it emits smaller code because it assumes it will not need to defensively place it into a Scalar container 17:09
18:07 reportable6 left 20:10 reportable6 joined
japhb jnthnwrthngtn: Oh that is very valuable information. I had been avoiding putting types on super-frequently-called methods under the assumption that there was extra cost there ... but it sounds like I may have been preventing inlining. 20:17
20:35 [Coke] left 20:39 [Coke] joined 22:13 sena_kun left 22:15 sena_kun joined
MasterDuke ugh, now dealing with windows build problems again. guess it's soon going to be time to break out that vm of mine instead of waiting for ci jobs to run... 22:26
especially since there still doesn't seem to be a way to specify the jobs to start first (windows and jvm, since those take the longest) 22:27
ok, appeared to have solved the mingw bcrypt problem. now just `inst-rakudo.exe.o:main.c:(.text+0x84): undefined reference to `mi_free'` on windows (mingw and msvc) to fix 23:03
[Coke] MasterDuke++ 23:04
MasterDuke it will be nice to get back to projects that don't involve the build system 23:06
[Coke] :) 23:07
MasterDuke oh, this last one may actually be a pretty trivial fix, let's see... 23:10
github.com/rakudo/rakudo/commit/1b5c0b9516 23:15
now to wait a couple min to restart the CI jobs in the mimalloc PR and hope they pull in that rakudo commit
ah ha! 23:36
Nicholas: do you mind testing github.com/MoarVM/MoarVM/pull/1634 on some of the crazier systems you have access to? 23:37