github.com/moarvm/moarvm | IRC logs at colabti.org/irclogger/irclogger_logs/moarvm
Set by AlexDaniel on 12 June 2018.
00:05 moon-child left 00:46 moon-child joined 01:26 lucasb left 02:36 raku-bridge1 joined, raku-bridge1 left, raku-bridge1 joined 02:37 raku-bridge left, raku-bridge1 is now known as raku-bridge 02:38 MasterDuke left 04:27 evalable6 left, linkable6 left 04:30 evalable6 joined, linkable6 joined 06:43 Altai-man joined 06:54 Altai-man left 09:38 domidumont joined
nine Good weekend, #moarvm! 10:01
nwc10 good *, #moarvm 10:32
10:38 domidumont left
nine So I sat down and typed in a systematic list of test cases for NativeCall memory management with functions like void take_string(char *str) {}; void take_and_free_string(char *str) {free(str);}; void take_simple_struct(SimpleStruct *s) {} 10:54
I.e. for strings, structs and arrays functions that take those as arguments or return them with either taking/giving ownership of the memory or not.
When calling those I hit the first snag with the second function that takes and frees a string. There's no way to declare this transfer of ownership. But at least there's a way around it with take_and_free_string(explicitly-manage('foo')); 10:56
I.e. every caller has to do that.
This only works for strings however. There's currently no way to do the same with structs (i.e. the fourth of 22 test cases). 10:57
Then there's char *return_allocated_string() which expects you to free() that string. I don't think there's a way to do that. 11:00
11:33 moon-child left, samcv left, earenndil joined 11:34 samcv joined
nine Nice. For strings we actually already have the infrastructure for communicating to the VM whether an argument must be freed by the VM or not. With this implementing sub take_and_free_string(Str $s is transferring-ownership) is native($lib) { !!! } was a rather trivial change to NativeCall.rakumod. 11:48
With that take_and_free_string('foo') no longer leads to a double-free and the caller doesn't have to remember using explicitly-manage
timotimo i take it you also have a "returns_pointer_into_middle_of_managed_buffer" test 11:50
or is that equivalent to "return a pointer that must not be freed by the vm"
nine the latter 11:54
Because it really comes down to the question of who is responsible for freeing some memory.
timotimo ok, how about this then: "returns_pointer_into_a_buffer_that_the_vm_manages" :P 11:56
can't do that without the GC being aware of the relationship
nine Isn't that just a pointer that the VM must not free()?
timotimo not only that
it's also that the other pointer must not be freed until this reference is no longer used 11:57
since in this hypothetical situation the VM is actually responsible for freeing the original buffer when it becomes garbage
nine Sounds like we can delegate the responsibility for that to the programmer. A typical situation where this occurs would e.g. be you have a string and push it into some parsing library which may return pointers into the original string. The docs will tell you that. And you are responsible for keeping the original string alive. 11:58
timotimo i guess that's fair 11:59
nine No magic on our side will be able to shield a user of a C library from having to think about memory management. 12:00
So far NativeCall mostly pretends to do so - and it does get far with that - but that leads directly to these situations we can't handle currently. 12:02
timotimo right 12:03
nine Same trick as for arguments also worked for returned strings. Now on to something more challenging: structs 12:15
timotimo watch out, or you'll be thunderstruct 12:22
nine that....hurt :D 12:23
timotimo hiiyeeah-yeaah
nine Hm...I guess if one passes a CStruct to a function that wants to take ownership, throwing an exception if we aren't actually owning that wrapped struct is appropriate. 12:41
timotimo anything about nested structs (by which i mean has, not HAS) 12:44
nine ? 12:45
timotimo if you have structs that point at other structs, we will require the programmer to handle ownership transfer? 12:46
oh btw, passing structs literally via parameters is supported by both dyncall and libffi, you "just" have to give it a representation of the struct's ... structure 12:47
i looked into doing it once, somehow didn't reach the goal
knowing me i was probably just not able to concentrate at that time?
the only library i've wanted to bind that uses structs passed by value into a function was SDL_ttf 12:48
jnthn wonders if `is transferring-ownership` is a little long, and ponders `is disowned` 12:58
nine As I mentioned when I first wrote about this issue, I'd hate to have to type "is transferring-ownership" all the time. So any better ideas are greatly appreciated :) 13:00
timotimo sad we can't use "will" here
jnthn More deeply, whether we really want to have the explicitly managed mechanism "survive", or if we'd be better off just attaching the semantics to the parameters, and then passing that info along as part of the signature spec 13:01
(The explicitly managed mechanism depends on a mixin to each thing we want to manage, and that gets costly)
I'm guessing the other situation is when we get memory back as a return value, and then need to deicde if we claim it or not 13:02
And another - horrors - is rw parameters :)
13:08 MasterDuke joined
MasterDuke sounds like moarvm needs some sort of memory lending verifier... 13:09
nine Returned structs are now handled correctly in both cases :) 13:22
sub return_simple_struct(--> SimpleStruct) is native($lib) { !!! }; sub return_allocated_simple_struct(--> SimpleStruct) is transferring-ownership is native($lib) { !!! } 13:23
jnthn: the great majority of cases will be covered by the parameter and routine traits. But there will still be cases where we have to mark individual objects. E.g. we get a pointer to a struct and are supposed to manage an individual field. Or inverse. 13:24
13:42 MasterDuke left 14:01 sena_kun joined 14:39 lucasb joined 14:56 domidumont joined 15:12 sena_kun left, sena_kun joined 15:30 Altai-man joined 15:32 sena_kun left
nine Strings are a bit of an unpleasent special case in NativeCall. CStruct, CArray and friends can easily contain a flag and manage (or not) the referenced native data structure. But strings are not even aware of their native counter part. So CStruct and friends need to keep track of their string's status 15:41
15:49 sena_kun joined
nine Next interesting bit: when we get a struct containing a pointer to another struct from a C function and it expects us to free both, there's a difference whether we actually access the contained struct in Raku code or not. If not, we never create a CStruct wrapper object which would handle freeing of that memory 16:01
So many things to fix... 16:02
16:34 sena_kun left 16:38 dogbert17 joined
lizmat or tests to write :-( 16:56
lizmat just broke %h<a b> = %h<b a> and spectest was silent :-(
nine Yeah, there are just so many interesting cases 16:58
18:36 zakharyas joined 19:12 moon-child joined 19:13 rba joined, earenndil left, nine left, nine joined 19:31 sena_kun joined 19:32 Altai-man left 19:55 patrickb joined 19:58 patrickb left 20:46 Geth joined
nine Before github.com/MoarVM/MoarVM/commit/e9...6c3d8b9329 MVM_nativecall_refresh was b0rked for inlined attributes. Now it's b0rked for normal ones :( 20:47
lizmat :-( 20:48
nine At least it pays off that I want to clean up our whole NativeCall memory management mess in one go instead of fixing one issue at a time. Once you're deep in there, it becomes easy to find the correct solution to such trivialities 20:52
Once I got that done I will come across the next interesting issue. A function like void take_and_free_struct_with_struct(StructWithStruct *s) { free(s->str); free(s); } will lead to a use-after-free in MVM_nativecall_refresh. 20:54
Thing is, once we give away ownership of some memory, we cannot even rely on that memory still being available at the end of a call. So just accessing it again unconditionally is no longer a winning strategy. 20:55
Probably better to just mark it dirty (good thing that there's one more bit available in the cstruct's address) and refresh it on the next access to an attribute (at which point it's the user's fault if it blows) 20:56
timotimo we also don't have any way to mark structs or other things "volatile"; would we need to have that? 21:12
vrurg .tell jnthn commaide.com certificate has expired. 21:43
tellable6 vrurg, I'll pass your message to jnthn
21:46 domidumont left
nine timotimo: well in a sense we treat those things as volatile by default (hence the call to MVM_nativecall_refresh). So we'd need the opposite 22:03
timotimo we don't nativecall_refresh on every access to attributes tho, right? 22:04
nine oh, that's true 22:05
22:05 sena_kun left 22:14 moon-child left 22:24 zakharyas left 22:27 moon-child joined 22:45 moon-child left 22:51 moon-child joined, moon-child left 22:52 moon-child joined 22:53 moon-child left 23:09 moon-child joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!