01:52 ilbot3 joined 04:47 ilbot3 joined, samcv joined, mst joined, samcv joined 04:48 mst joined 04:49 harrow joined 04:50 ggoebel joined, ilmari[m] joined 04:52 ingy1 joined 05:25 sivoais joined 06:00 nwc10 joined 06:04 Util joined 06:05 brrt joined
brrt good * #moarvm 06:15
.tell markmont thanks for your links and suggestions, i will read them 06:16
yoleaux brrt: I'll pass your message to markmont.
brrt .tell samcv i think that using the grapheme iterator for the text string of the index makes a *lot* of sense 06:22
yoleaux brrt: I'll pass your message to samcv.
brrt 10/10 would support that :-)
also, that thread, wow 06:37
bugzilla.mozilla.org/show_bug.cgi?id=506693
that's some wonderful clash of ego's 06:38
07:17 brrt joined 07:36 nine joined 07:50 domidumont joined 07:52 nine_ joined, zakharyas joined 08:03 ilbot3 joined 08:08 pmurias joined 08:15 Zoffix joined, ZofBot joined, buggable joined, leego joined 08:21 arnsholt joined 08:55 jnthn joined
jnthn moarning o/ 08:55
brrt moarning
jnthn Huh, odd, apparently I got K-Lined for spam overnight?
Oh, apparently along with a lot of people...
Geth MoarVM/atomics: 6693d7b5e5 | (Jonathan Worthington)++ | src/gc/worklist.h
Include bad pointer in an error.
 08:57
MoarVM/atomics: f50afea628 | (Jonathan Worthington)++ | src/core/frame.c
Extra GC debug checks after frame promotion.
jnthn Apparently I forgot to push those yesterday...
09:07 nebuchadnezzar joined
jnthn Currently looking into RT #131857 fwiw 09:08
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=131857
pmurias jnthn: moarning \o 09:16
09:17 avarab joined 09:18 avarab joined
Geth MoarVM: 767a8900ec | (Jonathan Worthington)++ | src/spesh/args.c
Blacklist unhandled param ops in spesh args.

It would be good to handle these, but not recognizing them as param related ops leads to generating specialized code that doesn't work. Fixes Rakudo RT #131857.		 09:19
synopsebot6 Link: rt.perl.org/rt3/Public/Bug/Display...?id=131857
jnthn o/ pmurias
10:11 leego joined
jnthn Merge time! 10:19
Geth MoarVM/master: 25 commits pushed by (Jonathan Worthington)++, ven++
review: github.com/MoarVM/MoarVM/compare/7...88c6851f51		 10:20
10:25 ilmari[m] joined
timotimo my socket peer/host patch doesn't build on windows, perhaps missing a header ni the ifdef windows section 10:29
jnthn Yeah, that was my guess to
*too
timotimo looks for appveyor for moarvm
ah, i have one under timo/moarvm 10:30
oh, it's cool that we compile our code before the 3rdparty code 10:32
google "windows in_port_t" ? "How to Import Photos with Windows 10 - dummies"
10:34 brrt joined
brrt K-lined? 10:37
timotimo yeah, a whole lot of folks got that 10:38
nwc10 "me too" 10:39
jnthn: what fun and excitement does this merge bring us?
jnthn nwc10: Support for CPU-backed atomic operations in Perl 6
Plus a couple of fixes for places that didn't uphold GC invariants 10:40
timotimo so ... i can probably just use an unsigned long to store the port? ports are limited to what was it 16 bits? 10:42
rather than use in_port_t
that could be the only thing missing on windows
jnthn timotimo: Yeah, I'd expect so
timotimo let's see what appveyor thinks 10:44
geth got the boot? 10:45
no, geth is still connected 10:46
jnthn Geth is just being a bit slow
Geth MoarVM: a0cc6d301d | (Timo Paulssen)++ | src/io/asyncsocket.c
work around missing in_port_t on windows

by using a sufficiently big integer™ instead.
timotimo OK, that's fair
it does get past asyncsocket.c now 10:49
the appveyor build just stopped after a whole lot of nqp tests and isn't signalling success 10:58
jnthn lunch & 11:27
11:30 cognominal joined
dogbert17 jnthn: I made a small update to github.com/MoarVM/MoarVM/issues/554 11:31
11:36 ilbot3 joined 11:48 markmont joined 11:58 ilbot3 joined 12:15 zakharyas joined
brrt ohai markmont 12:20
i just replied to your last comment :-)
let me know what you think :-)
interstingly, the paper you linked argues for using a separate process for code generation 12:21
that's a pretty cute idea, i think, but to say 'lets add RPC and *then* it'll be safe, well, thats additional surface area i think 12:22
markmont brrt: hi! Your points are all good ones. I would not want to do RPC at all, that is way too much overhead and complexity. 12:23
yoleaux 06:16Z <brrt> markmont: thanks for your links and suggestions, i will read them
markmont I'm looking at this pragmatically: The dual-mapping "solution", despite having a lot of valid points against it, is nonetheless what both RedHat and the NetBSD folks are going with. My personal opinion is that it is better to get on board with their solution than it is to add exceptions to their security policy, but, like I said, this is just my opinion. 12:25
brrt that's fair 12:26
i think we don't need to dualmap
we can have just a single mapping that is RX
since we have a FD, we can write to it in any other way we like
markmont Also, from a pragmatic viewpoint, without this MoarVM patch to implement W^X for JIT, MoarVM is *already* calling ffi_closure_alloc and doing the dual mapping, if it is compiled with --has-libffi and is on a platform that causes libffi to enable the dual mapping.
jnthn ooc, what happens if we want to tricks like patching the machine code once it's already running?
brrt sure, that is also true
you'll have to go past me :-P 12:27
nine brrt: "Furthermore, if I generate a private file with mkstemp as given in [this example](www.akkadia.org/drepper/selinux-mem.html) then I can not prevent another process from opening the file before I unlink it"
brrt: you actually can using O_TMPFILE. The temporary file will never be accessible in the file system.
brrt nine, that's true, and libffi does that on platforms that support it
jnthn: i'd argue for patching the spesh table before patching the assembly language 12:28
there will be just a few cases in which that won't be enough
markmont Here is a question I haven't asked yet: why does --has-libffi exist for MoarVM? What problem does/did it solve? 12:29
brrt it probably works in some context that dyncall doesn't, and vice versa
honestly, i don't exactly know
jnthn It was pretty much that
markmont If we're going to simplify, I find it appealing to drop dyncall and get libffi working everywhere and fully use the functionality it provides, including ffi_alloc_closure. 12:31
brrt also, re: libffi doing something i find questionable, that's their responsibility (and I wouldn't have known if we weren't looking into it)
markmont Otherwise, it seems odd to have it but be reluctant to fully leverage it. Again, just my opinion because the dual-mapping doesn't bother me.
brrt i don't think libffi's support is that broad just yet
jnthn I think somebody looked into it and found that they had non-overlapping bits of platform support. 12:32
brrt
.oO( if only we know what bits these were )
jnthn It's been a while though
As to dealing with W^X stuff, I think if various platforms are going there, we'll just have to grin and bear it, whatever we think of it. 12:34
markmont In terms of libffi's support not being broad, it's used by the core pieces (not packages/modules for) Dalvik (Android), OpenJDK, and Python.
jnthn: I am fully in agreement with grinning and bearing it. But, to be full-disclosure, both Google V8 and PHP have said "no way in hell". 12:35
Google V8 is significant because it directly affects NodeJS.
jnthn Well, v8 I think *does* do patching of the generated machine code 12:36
markmont Every time I have to install NodeJS on one of my boxes I wind up cursing and jumping through hoops to grant special execmem permissions just for it... but I do it.
nine markmont: OOC what does Python use it for? Accessing C libraries in CPython works pretty much the same way as in Perl 5.
markmont sourceware.org/libffi/ says it is used for ctypes.
jnthn Which means it'd need to be able to do W after X 12:37
I guess if it's two mappings of the same memory that works though?
brrt aye
jnthn But if it doesn't work out then it'd be a real blocker for them
And wanting to do that is quite a reasonable thing for a modern VM to wish to do
Anyway, I'm +1 to not making people jump through hoops of changing security settings. I'm OK if the solution is "you need to build with libffi" for now 12:39
I'm not too fond of #ifdef in a few places in the JIT code though
markmont If we keep libffi but don't want to use ffi_closure_alloc in the JIT compiler, I'd like to see if we can find a way to avoid calling it in the nativecall callback code. This is separate from whether we support W^X out of the box or not. It just seems... odd... to say that dual mapping is OK in one place but not the other.
jnthn And would prefer a better designed abstraction for that 12:40
brrt i'm okay with using ffi_closure_alloc, but like jnthn, better abstraction, and explicit enabling
jnthn So that if we want to in-source the mechanism rather than deferring to libffi some day, then we can
brrt nods
markmont Yes, that sounds good.
brrt cool :-) 12:45
markmont brrt, jnthn: If everyone is OK with it, I'll close the current PR and create a new one that encapsulates everything through a clean API and adds either a Configure.pl flag to enable it, an environment variable, or both, depending on what you suggest. 12:47
samcv markmont++ 12:48
yoleaux 06:22Z <brrt> samcv: i think that using the grapheme iterator for the text string of the index makes a *lot* of sense
samcv good morning everyone. well sort of morning. i woke up early 12:49
brrt markmont++ thanks a bundle :-)
good * samcv 12:50
samcv markmont, i didn't comment on your PR but i saw them and wanted to thank you on the work you're doing on libffi 12:51
markmont Regarding libffi vs dyncall, I just learned that while dyncall works under MS Windows, you have to use either Cygwin or MingGW to compile it there -- it doesn't support the Microsoft development environment. libffi does support the Microsoft toolchain, though.
samcv and the other questions you're looking into as well.
jnthn markmont: Yeah, I recall that. That's not an acceptable build requirement for us. 12:52
*recall that now 12:53
Yes, new PR as you describe would work for me, and will be happy to merge such a PR
Thanks for working on this.
markmont samcv: thanks! 12:54
jnthn: it will likely take me a few days to a week to do since I want to do it right and will be using personal time to do it. But I'll post design proposals and intermediate versions of the code here for comment and feedback as I go. 12:55
jnthn That's fine, take the time you need :) 12:56
samcv :)
markmont Since we're conversing rather than coding right now, any thoughts on github.com/MoarVM/MoarVM/pull/633 ? 12:57
Hopefully it is a straightforward bug fix for callbacks when MoarVM is built with --has-libffi.
The summary is that we can't free the call interface (cfi) at the end of the callback handler, since it is used as a part of the closure. Freeing it makes the closure, and hence the callback, not callable again. 12:59
Since the closure never gets GC'd, it seems safe to never free cif.
jnthn Ah, right 13:00
huh, something thought I'd merged that. I certainly read it and thought it made sense :)
Geth MoarVM: ceedef892a | (Mark Montague)++ | src/core/nativecall_libffi.c
Make sure the libffi callback can be reused.

Rakudo test t/04-nativecall/21-callback-other-thread.t tests 2..9 fail with a segmentation fault or glibc error (double free or malloc() problem) when MoarVM is built with --has-libffi.
The problem is due to the callback handler freeing the memory for its first argument (the libffi call interface, cif). Doing this makes the callback not reusable (the callback still exists in tc->native_callback_cache).
MoarVM: 67fc27b6c4 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/core/nativecall_libffi.c
Merge pull request #633 from markmont/libffi-callback-other-thread

Make sure libffi callbacks can be reused.
markmont Thanks jnthn++ !
jnthn s/something/somehow/ :) 13:01
Thanks markmont++ for the patch
:)
samcv markmont++
markmont, you said it fixes one of the tests? is that in roast or in the rakudo test?
`make test` in rakudo directory as it were
markmont Yes, it fixes one of the Rakudo `make test` tests, t/04-nativecall/21-callback-other-thread.t 13:02
brrt markmont++ indeed :-)
markmont The test is not from Rakudo `make spectest`.
samcv nice :) 13:03
yeah i know with libffi some of the fail so i'm glad someone is working on this. at least gentoo compiles with --with-libffi
i think the other distros do not 13:04
and then sounds like that's what is needed on bsd as well
i think gentoo just used it since it was a library they already had in their tree, and they always like to use local versions of libraries if possible
markmont No, NetBSD 8 will work fine without libffi, if you run `paxctl +M /path/to/bin/moar`. And the other BSDs will work fine without libffi even without this. 13:05
brrt you can make selinux do the same thing, of course :-) 13:06
but anyway. i agree with jnthn that not having to jump through hoops is the preferable bit
markmont Well, the SELinux deny_execmem boolean defaults to "off" under Fedora and RHEL, currently. So for SELinux it's a question of making MoarVM work when a local administrator has set deny_execmem=on, as I do. 13:08
In that case, you can write a local SELinux policy to permit the mprotect() call only for MoarVM, but that sort of goes against the reason for turning deny_exemem on in the first place. 13:10
brrt what about the chcon bit in the firefox thread?
markmont That was removed a few years ago as a part of tightening things up and making it harder for code the SELinux people disagree with to work. The chcon command won't give an error, but execmem_exec_t is now an alias for bin_t. 13:14
brrt oh, that's agreeable of them 13:15
14:11 pharv_ joined 14:12 lizmat joined
samcv jnthn, do we have a function which moves the gi to a position and then gets the grapheme without initializig the grapheme iterator? so it uses the supplied iterator? 14:46
and then we don't have to init the iterator each time. i don't think we have it, but i'll add it and we can easily replace everywhere that uses get_grapheme_at_nocheck with that
brrt MVM_string_gi_move_to 14:47
?
src/strings/iter.h
samcv no i know that one
MVM_string_gi_get_grapheme_at << looking for a function like this but i don't think it exists 14:48
that moves it to the position in the grapheme iterator and then returns the requested grapheme
jnthn samcv: So far as I rememeber move_to assumes "from the start" or so 14:49
samcv i mean get_grapheme_at_nocheck initializes a brand new grapheme iterator and then moves it then returns it. but i don't think we have one that takes a grapheme iterator
jnthn So we are missing what you'd wanting, I think
samcv no that is fine
the algorithm i have in works off get_grapheme_at_nocheck currently so
i don't need to move it X spaces, i'm just trying to save having to create a new grapheme iterator for every codepoint we grab 14:50
jnthn So you just want a function that does something like move_to + get? 14:53
samcv yeah
jnthn Sounds sensible, but agree we don't have that yet, afaik 14:54
samcv i can make one though
ok cool
damn i'm getting iteration past end of grapheme iterator 14:57
hm will figure it out 14:58
15:06 brrt joined 15:12 AlexDaniel joined 15:16 JimmyZ joined 15:25 TimToady joined
samcv ok anyway gonna go back to the knuth_morris_pratt branch and throw away the new code for a sec. turns out if i try and compile nqp with it i get a segmentation fault. even though i can pass spectest fine 15:29
jnthn, how do i debug nqp compilation seg faults?
Geth MoarVM: 8e7b68ebf9 | (Jonathan Worthington)++ | 11 files
Don't guardsf results of multi resolutions.

We aren't smart enough to handle this case yet, so it caused a ton of misses and thus deoptimizations. This fixes a performance loss in a bunch of programs from invocation speculation (so improves the Text::CSV benchmark again, probably back to its best), while retaining the performance gain for things that won from it (such as the million line reading benchmark).
jnthn samcv: Build a debug MoarVM, copy the moar invocation that fails, and then valgrind <paste> :) 15:30
15:30 Geth joined
jnthn Alternatively with gdb, just gdb /path/to/moar and then in it r <paste the args> 15:30
The NQP build invokes MoarVM directly so can just nab the args as they are
8e7b68ebf9 should get our sub-4s test-t back 15:32
timotimo i much prefer "gdb --args /blah/moar foo bar baz" 15:38
jnthn ah, right :) 15:39
samcv ah ok 15:41
i'll try valgrind first
Access not within mapped region at address 0x20 15:42
oh no
must be trying to get past the end
jnthn Yeah, that doesn't look like a pointer to me :)
If anything it looks like a null deref
samcv at least i can see where it is. i had this problem in the pattern processing. probably have to add a check in to the other part too 15:43
yay fixed it 15:46
very good
though i'm not sure how it gets that way. i added exception throws after every place pat_offset is changed. and it doesn't throw 15:55
Geth MoarVM: 8f759a6185 | (Jonathan Worthington)++ | 4 files
Smaller spesh log quota for spawned threads.

To help reduce memory usage a little when we spawn a lot of them.
MoarVM: 2455e81851 | MasterDuke17++ (committed using GitHub Web editor) | src/6model/reprs/VMArray.c
Remove impossible check

n is an MVMuint64, so it can never be negative.		 16:02
MoarVM: e5d823b06b | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/6model/reprs/VMArray.c
Merge pull request #646 from MasterDuke17/patch-2

Remove impossible check
MoarVM: 737e89c4de | MasterDuke17++ (committed using GitHub Web editor) | src/6model/reprs/P6opaque.c
Remove impossible check

slot is a size_t, so it can never be negative.
MoarVM: c4ee23b4cf | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/6model/reprs/P6opaque.c
Merge pull request #647 from MasterDuke17/patch-3

Remove impossible check
samcv jnthn, this is what i'm thinking of for the MM_string_gi_get_grapheme_at gist.github.com/samcv/49edd75afe6f...605d99e6b0 16:05
so it's basically the same as the get_grapheme_at_nocheck except it just uses the specified grapheme iterator you give it
maybe needs another name idk. cause it doesn't do anything with the grapheme iterator except for strands, to avoid doing work 16:06
jnthn And it works on multiple calls?
Time to go and cook dinner; bbl 16:10
samcv it doesn't work :( 16:11
argh
does move_to go off of offset then? 16:12
from last position?
so it assumes that it's starting from 0?
oh. hahhaha 16:18
Moving to index 0 of string 0x7fb1c3757240
Iteration past end of grapheme iterator
XD
16:30 [Coke] joined 16:57 dogbert2 joined 17:18 AlexDaniel joined 17:19 zakharyas joined 17:44 domidumont joined 18:20 lizmat joined 19:38 zakharyas joined 19:40 raschipi joined 19:47 brrt joined
brrt good * 19:47
timotimo good 19:51
samcv good * 20:16
20:17 brrt joined
nine One a block is JITed, it won't get changed anymore, will it? So something like "if (MVM_spesh_log_is_logging(tc)) MVM_spesh_log_parameter(tc, arg_idx, param);" won't make sense anymore, will it? 20:41
samcv MVM_string_gi_move_to totally doesn't work at all apparently 20:43
except for right after MVM_string_gi_init
it appears to have absolutely 0 effect on any of the values
hmm
maybe it's relative? ugh 20:44
anyway moving onto something else. so i did some statistics doing nqp::index('aaaaaaaaaahellothere', 'hellothere', 0) 21:10
i wrote a function to cache the last graphemes value if it requested the same one a second time. and record hits and misses 21:11
H hit 0 miss 20; n hit 10 miss 10; so it looks like caching the needle if the needle is a strand or repetition could be useful
well any type of strand
and the haystack moves one more each time so the haystack could benefit from saving the grapheme iterator and only reinitializing it if the next requested codepoint is different than the one it previously asked for 21:14
and it would be faster to only do these things when that storage type was a strand, otherwise there's not too much benefit 21:15
well by that i mean it will be wasteful when we can just get the blob data immediately
jnthn nine: Correct, that's why the JIT doesn't bother emitting that part 21:25
nine: Also why we have things like sp_decont which is decont without the logging, to remove the overhead some even when interpreting spesh'd bytecdoe
samcv: Yes, that's the semantics I *thought* move_to had 21:26
samcv relative move?
jnthn samcv: Which is why I was asking if you expected it to handle relative movement
Well, the semantics today are "aboslute distance from the start of the string" 21:27
samcv and it can't go back right?
jnthn Maybe we were using relative in different ways :)
samcv yeah it doesn't though
i mean i pull a grapheme, then move to 0
then grab again and get the next grapheme after the previous one (not 0)
so i assume it is asking for relative change or is broken
jnthn Right, it was only used so far for the case where we call it after init 21:28
samcv yeah
jnthn No, you can't go backwards for certain
samcv yeah
jnthn It's an iterator
But I thought since you were scanning through a string you'd have relative offsets
And I thought it might do something sensible with those
samcv so it seems to be relative. though asking it for some other thing seemed to cause weirdness. or it may have been my code
yeah. that will take some extra work to do 21:29
jnthn It's certainly the case that it was only built for use after init though :)
But I think I misunderstood what you wanted to achieve, so I suspect my answers about it only confused stuff more...
samcv because currently i can't get the same codepoint again (move 0)
becuase the algorithm never backtracks, but it does request the same again
jnthn Ah
Yeah, asking for a grapheme moves the iterator 21:30
You'd need a separate operation that gets current without moving
samcv H hit 0 miss 20; n hit 10 miss 10; when i cached the previously requested grapheme
and haystack moves forward 1 much of the time and sometimes moves more 21:31
dogbert2 timotimo: any theories wrt this ASAN barfage? gist.github.com/dogbert17/cb6b2556...82393f654f
samcv with needle requesting the same codepoint it requested the last time 1/2 of all requests
so i got some very useful data
21:42 lizmat joined
timotimo dogbert2: looks like the same thing you found last time you reported something related to "join" 21:46
i.e. spesh worker thread accessing a piece of a tc that's already been freed 21:47
dogbert2 timotimo: aha interesting, perhaps I should try it with MVM_SPESH_DISABLE=1 as well then 21:56
it fails with that flag set as well albeit slightly differenly 21:59
22:03 MasterDuke joined
dogbert2 timotimo: I updated the gist with the second failure 22:05
22:43 markmont joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!