github.com/moarvm/moarvm | IRC logs at colabti.org/irclogger/irclogger_logs/moarvm
Set by AlexDaniel on 12 June 2018.
00:03 sena_kun joined 00:05 Altai-man_ left 02:02 Altai-man_ joined 02:05 sena_kun left 02:32 lizmat joined
timotimo apparently using a react/whenever on a process when tapping the stdout will not do backpressure on the process itself 02:52
i tried with a "sleep" in the "whenever $process.stdout" block, but the memory growth was many times as much as the amount of characters it said it "got"
sleep 2, to be exact 02:53
not infinite sleep
i thought we had this whole "permit" system for i/o that does backpressure on this level?
hum, so the default permit value is -1, unless you call nqp::permit on the AsyncTask? 03:05
looks kind of like the only usages of permit are with -1 on the third argument, i.e. turn something on at full blast (this is searching for nqp..permit in all of rakudo) 03:09
oh, if i use permit with a number i'll have to permit it again manually so that stuff continues happening after the initial permit amount is drained? i guess that makes more sense than just renewing the permit when it's somehow magically determined that a work item has been processed 03:20
have i been looking at a far outdated generated core setting? 03:40
perhaps not too outdated 03:41
04:03 sena_kun joined 04:05 Altai-man_ left
timotimo i'm now "shit posting" a patched rakudo that permits the process to write 128 times every 1 second to see if that works well enough to keep memory usage low in this workload 04:14
ideally it'd start with a lot and go down to a little, so that it can spawn its maximum threads early and then have a constant trickle of chunks for new threads to be started with 04:15
i can literally implement that lol 04:17
06:02 Altai-man_ joined 06:05 sena_kun left 07:34 tobs joined 07:43 zakharyas joined
nwc10 good *, #moarvm 07:47
I am temporarily in a deeper (longer) hole than usual
08:03 sena_kun joined 08:05 Altai-man_ left 09:22 squashable6 left 09:24 squashable6 joined
jnthn nwc10: Gotthard base tunnel? 09:59
10:02 Altai-man_ joined
lizmat jnthn: github.com/rakudo/rakudo/blob/mast...rs.pm6#L92 10:03
*every* time something does "but True", it will run this code and create a new type
every single time 10:04
a new type
that can't be right ?
10:05 sena_kun left
jnthn lizmat: That's how it reads to me. 10:10
lizmat: There's surely a more optimal way to do that :)
lizmat yeah
I was thinking of making trying to but an non-value type fatal 10:11
and cache the roles on the .WHICH of the value
jnthn I think this code has been around for a long while, and maybe predates 6pe (6model parametric extensions), which provides a mechanism for caching these things.
No, don't build your own cache.
Because it will probably get precomp wrong 10:12
(6pe knows how to intern them across compilation units)
I'm not sure that we should cache it on value though 10:13
Well, actually, for enum types that makes sense
For for Int, there are a lot of integers ;)
lizmat well, it closes over the value given
jnthn I think maybe the best way to do it is that the role brings in an attribute to hold the value 10:14
lizmat well, creating a new type every time, is not only memory hungry, it also eats CPU like crazy
jnthn And then we create one role per type that is mixed in
lizmat loop { my $a; $a but True } eats about 20MB / sec on my mechine
jnthn one for Bool, one for Int, etc.
lizmat I hear what you're saying, yet I have no clue on how to move forward on this :-( 10:20
jnthn Look at how the mixin cache works. You need 10:24
1. some type to serve as the "base" for the caching
2. to have a parameterizer that builds the role based on the type (I think the simplest way is to have an attribute with the name of the type) 10:25
3. Then use the (iirc) parameterizetype op to obtain the produced role for the type (which will use a cached one if possible) 10:26
4. Mix that role in, and you can probably use the whole BUILD_LEAST_DERIVED thingy to set the attribute
lizmat ok, let me try to grok that 10:27
10:50 patrickb joined 11:28 zakharyas left 11:45 AlexDaniel joined, AlexDaniel left, AlexDaniel joined 12:03 sena_kun joined 12:05 Altai-man_ left 12:57 MasterDuke left 13:08 MasterDuke joined 13:34 zakharyas joined
timotimo media.ccc.de/v/ASG2019-127-gnu-pok...inary-data 13:37
"GNU poke is a new interactive editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them. Once a user has defined a structure for binary data (usually matching some file format) she can search, inspect, create, shuffle and modify abstract
entities such as ELF relocations, MP3 tags, DWARF expressions, partition table entries, and so on, with primitives resembling simple editing of bits and bytes. The program comes with a library of already written descriptions (or "pickles" in poke parlance) for many binary formats."
13:55 Kaiepi left
[Coke] ... wow, I haven't heard "pickle" since the late 90s when I was writing code in Modula-3. 13:55
13:56 Kaiepi joined
timotimo well, python has pickle 13:58
14:02 Altai-man_ joined 14:04 sena_kun left
[Coke] as you now know, I am not a python guy. :) 14:08
timotimo :) 14:09
why can't we unpickle python-pickled data into raku datastructures yet
including code because i have no other hobbies 14:10
nwc10 jnthn: no, merely the Lainzer Tunnel) 14:25
but the train wifi worked way better than I assumed
[Coke] nwc10: heh. whenever I used to take the train from Albany to NYC, I would have to bring a book the wifi (and cell coverage) was so bad. 14:27
... fixed by just never leaving the house these days. :|
nwc10 aargh, this is a new one on me - this train would fail the slop test 14:32
(ie does it spill your cup of tea)
it is vibrating the table
(this is not jolts caused by poor track, which is the usual intent of the "slop test" joke. But also about the laziest way to inspect the track) 14:33
jnthn Hm, what's the source of the vibration? 14:34
nwc10 I suspect that something has chanced on the resonant frequency of the seat assembly 14:35
from master/master/master, is this an error I'd currently expect to see from Rakudo: New type Perl6::Metamodel::CurriedRoleHOW for Perl6::Metamodel::ConcreteRoleHOW is not a mixin type 14:36
ie have I messed up my build trees, or is it a temporary LTA upstream ?
14:50 patrickb left 14:51 patrickb joined
nwc10 what is this heresy? Surely emacs is sufficient to edit *anything*? :-) 14:58
jnthn nwc10: Where do you see it? It's not familiar, but I'm a bit back from HEAD thanks to being in branches nearly all the time these days... 15:00
nwc10 jnthn: I think that I might have goofed the NQP build (ie the wrong build tree, or just the wrong install target) 15:01
15:14 patrickb left 15:38 MasterDuke left 15:59 MasterDuke joined 16:03 sena_kun joined 16:04 Altai-man_ left 16:13 Kaiepi left 16:14 Kaiepi joined 18:02 Altai-man_ joined 18:04 zakharyas left 18:05 sena_kun left 18:34 Kaiepi left, Kaiepi joined 18:45 squashable6 left 18:49 squashable6 joined 18:56 [Coke] left
nwc10 jnthn: I think that this is likely to trigger a discussion that will have to happen on another day. 19:07
What I know (from adding a whole lot of assertions, etc) is that this code:
./rakudo-m
-Ilib -e 'my $m = Mix.new-from-pairs("a" => -20, "b" => 1.5); my $a = $m.Bag; my $b = $m.BagHash'
(which is part of t/spec/S02-types/mix.t
Is performing a hash delete while an iteration is in progress 19:08
and, is there a Raku language level spec on whether this is allowed, and hence needs to be supported?
it *happens* to work currently
but is it supposed to?
no, I shall be more eaxcting 19:09
it happens to work currently on MoarVM
I have no idea about other backends
as to the other thing I was confused about - problem was me. One build tree (NQP I think) was not at the master I thought that it was 19:10
MasterDuke what is it deleting? 19:15
nwc10 "a" => -20 I guess
and I must be tired
I have a backtrace from oops
at SETTING::src/core.c/Mixy.pm6:119 (/home/ghdev/Perl/rakudo4/blib/CORE.c.setting.moarvm:BAGGIFY) 19:16
also whether it hit oops or not depends on hash randomisation
MasterDuke doh
nwc10 you're going to need to make a few more d'ohs before you catch up with me :-) 19:17
or is that :-(
jnthn: OK, now I've realised my d'oh
and read the code
it's doing exactly the Perl 5 permitted operation - deleting the key at the iterator 19:18
(and then advancing the iterator)
so, um, er, erk. ish 19:19
the *exact* operation is probably viable, if it's done as a "method" on the iterator, not the hash 19:20
but I have no idea if Java, JS or anything else "we" (er, or them, over there on #raku) would like to target can support this 19:22
looking that code in Mixy.pm6, I think that it could be re-written to stack up the deletes in an array, and then loop over the array to perform them, only after the entire hash has been iterated over 19:23
set_difference.pm6 is doing the same (dangerous) pattern "at" line 122 19:25
and SETTING::src/core.c/Rakudo/QuantHash.pm6:952 19:27
lizmat nwc10: sure it can 19:28
I wrote it the way it is now *because* I checked with jnthn at the time, to know for sure it is ok to delete the key currently at the iterator
nwc10 OK. That is an implementation detail 19:29
lizmat should we decide that is no longer a good thing, then I'll rewrite the code accordingly
nwc10 that is an implementation detail that I wasn't aware of as being needed
and
lizmat if we can make hashes faster by *not* allowing that
nwc10 well, actually
lizmat by all means... go for it, I'll make the necessary changes
nwc10 I tweaked the implenmentation such that it can do *exactly* this delete without problems
but my (sort of problem) is that I can't validate that this is what is happening 19:30
lizmat ?
nwc10 and every other sort of delete is "potentially dangerous" (I think that currently my new code could oops)
acutally, with the current MoarVM master
any other delete, at the same time as an iterator exists
could lead to a "read from freed memory" 19:31
I might need to think about this - I've made plenty of mistakes already today - that "it could do X" might be another 19:32
19:32 SmokeMachine joined
lizmat nwc10: you mean in a single thread ? 19:33
or multi-threaded ?
nwc10 in a single thread
19:33 kawaii joined
nwc10 if the high level code happens to have two different iterators on the same hash 19:33
lizmat am pretty sure that would not happen in a single thread, unless people started doing their own nqp::iterator() calls 19:35
from a HLL point of view
nwc10 OK
I don't actually know the HLL stuff here :-)
I'm starting at the bottom
lizmat well... the iterator gets wrapped in a Seq
nwc10 but MVMIter holds a pointer
oops
MVMIter as implemented currently holds pointers direct to C memory "owned" by the hash 19:36
lizmat so technically, you *could* do something like %hash.grep: { %hash.keys }
nwc10 and if there are two iterators, and a hash access to a key found via the first (or anything else) causes the hash to change
then the second MVMIter could be left with a pointer to freed memory 19:37
lizmat or worse: %hash.grep: { %hash<foo>:delete }
so, even if the core would ensure only a single iterator / hash at a time, HLL user code *could* circumvent that 19:38
perhaps make it fatal to have 2 iterators on a hash at the same time ? 19:39
nwc10 that sounds awkward, and that really isn't the danger here.
it's sort of
1) we can cope with deleting the key that any (or all) iterators are currently pointing to
but
2) we can't "cope" with deleting any other key - ie
right now we could SEGV or worse 19:40
the code I've written would oops
and whilst I could make it not oops
that might hide other bugs - the oops was supposed to be the "how on Earth did you even get here - is MoarVM buggy?"
and
3) We don't have a way of finding the active iterators belonging to a hash 19:41
meaning that
4) I can't see a way of "enforcing" sanity - where sanity is "deletes are fine on hashes while iterators exist, providing either you only delete at the current iterator position, or you never read again from the iterator once the delete happened"
I think
the NQP code as written in the setting actually works just fine with the current implementation 19:42
and my new hash (as of two days ago)
but I don't know how to spot the "OK" case
lizmat as in: ok to delete the current ? 19:43
nwc10 yes. it is OK to delete the current
(Perl 5 neesd this, and I figured out how to make that work, by reversing the order that the internal hash metadata is traversed)
but I don't know how to fix my "paranoia" test code to be happy with this delete 19:44
lizmat would it make sense to have an iterator walk all of the hash and create a fixed list ?
nwc10 I think "no" because that would be expensive on memory
a slightly cheaper option is a way to be able to actually track active iterators 19:45
circular linked list of them
but this would then need a C API for "I'm releasing this iterator" and some code added to MoarVM
lizmat why not set a flag on the hash if there is an iterator active, and as soon as it has produce the final value, reser it? 19:46
nwc10 that was the point 3 above - We don't have a way of finding the active iterators belonging to a hash
lizmat with an option to resetting the flag and disabling the iterator
nwc10 I don't think that that solves this one, and I'd sort of implemented that already in my branch by having the iterators go in reverse, so that the end index is 0 19:47
as in zero
not "current size of hash"
lizmat if there would be only one active iterator allowed, then you wouldn't need to keep track of which iterators are active on a hash?
nwc10 because zero is the same everywhere :-)
lizmat hmmm
nwc10 "only one active iterator allowed" - welcome to Perl 5
and this is seen as a deficiency
lizmat well, maybe there is a reason for that :-)
maybe we should try to get some inspiration on recent hashing research ? 19:49
jnthn The other thing to keep in mind is that it's legal for multiple threads to iterate the same hash (so long as they don't modify it, of course) 19:51
So any "know the iterators" scheme would entail locking, and cost
Well, the cost of the locking
nwc10 All the stuff I've found so far as been about hash tables and how to implement them, has been about insert, delete, and fetch (on elements that are there, and elements that are missing)
not iterators
jnthn: gah yes.
currently I'm just wondering about removing my paranoia check for "at end" 19:52
having validated that all the places that hit it are actually "safe" because they delete at the current iterator
19:52 tbrowder joined
nwc10 and I made that work 19:52
lizmat but we're still not safe with an iterator in another thread when changes are made to the hash, right ? 19:53
nwc10 correct
and I think that we're not safe with an iterator in the same thread, and changes are made to the hash 19:54
(both current implementaion, and my proposed new one)
lizmat but the new one needs a lot less memory and is faster ? 19:55
nwc10 er, I've not yet got to the "faster" bit. I was trying to make it work at all first
lizmat Ah, I see
nwc10 it should be using less memory
lizmat but that's the goal, is it not ?
nwc10 I admit
if it's not measurably faster, I will be disappointed
lizmat been there, I hope you will not be disappointed 19:56
actually, at this point in time, I'm not sure what I'd want more: safe hashes that will never SEGV when being changed from different threads 20:02
20:03 sena_kun joined 20:04 Altai-man_ left
nwc10 OK, because I am thoroughly paranoid, and investigated all 15 failures... 20:07
I find that this one:
./rakudo-m -Ilib -e 'my %h; my $i = 0; for "a".."z" { %h{$_} = ++$i; }; my @all = %h{ %h.keys }; %h{*}:delete; ' 20:08
is a problem (ish)
at the C level it's calling "next" when the hash has been modified somehow
I need to dig further on this one 20:09
jnthn Hm, why is that one a problem? The assignment into @all is eager...
nwc10 but it doesn't seem to be "deleting at the current iterator position"
I don't know why it's a problem
I'm tired - it seems that today's travelling took more out of me that I expected
jnthn OK, it just feels like a "nothing should be awkward here" 20:10
(Unless there's something else surprising going on.)
nwc10 Unlike Perl 5, I have no feel for "how does the language map down to API calls"
I have no idea. Figuring out the "surprising" I think is left as a task for another day
which won't be tomorrow
jnthn Ohhh...maybe actually it golfs down to just the %h{*}:delete
nwc10 (unless plans chance)
change
jnthn Which I guess maybe does iterate all of the keys and delete them, if it's naively implemented 20:11
nwc10 I can't answer that
it has to be differnt in some way, as it's hitting my paranioa oops in MVM_str_hash_next 20:12
(which is what replaces HASH_ITER_NEXT_ITEM)
so it will be the case MVM_ITER_MODE_HASH in shift in MVMIter.c 20:13
(let's go check that)
yes, location equivalent to the current line 111 of MVMIter.c 20:15
and the Internet confirms my recolection that 111 is considered to be unlucky, but in far more detail than I ever thought possible: en.wikipedia.org/wiki/Nelson_(cricket) 20:17
lizmat nwc10: and this is from %h{*}:delete ?
nwc10 yes. My `say "yes"`; got called 20:18
no, I don't know why
but something about `%h{*}:delete` is implemented in a way that ends up calling C APIs in orders that are (I guess) LTA 20:19
lizmat nwc10: let me get back to that one, ok
nwc10 but I think that I should stop digging and go to bed
lizmat What I see, is this: 20:20
multi sub postcircumfix:<{ }>(\SELF, Whatever, Bool() :$delete!, *%other) is raw {
SLICE_MORE_HASH( SELF, SELF.keys.list, 'delete', $delete, %other );
}
which tells me it is passing a list of keys to be deleted
jnthn Looks like, yes 20:21
Probably not a terribly common construct to rush to optimize, but almost certainly there's a win to be had there 20:22
lizmat $more.cache.flatmap( { SELF.DELETE-KEY($_) } ).eager.list;
nwc10 jnthn: I think that I need to "temporarily" go "AFK" (ie sleep) 20:23
lizmat nwc10:: good night, sleep well!
jnthn nwc10: Rest well 20:24
nwc10 but in the future I need to go digging here - what is that Raku code turning into, and why is it calling MoarVM APIs in a strange order that doesn't seem to reflect what it ought to be doing (even if it could be written better)
and thanks both of you (and MasterDuke) for being awake with feedback at a useful time 20:25
lizmat nwc10: perhaps an API for resetting the hash, aka setelems(0), would be useful
the big issue with deleting all keys from a hash, is that the values should be returned as well 20:26
and that currently requires an atkey
if a deletekey would also return its value, then the HLL code could be much simplified
and return nqp::null if it doesn't exist, like with atkey 20:27
20:28 chansen_ joined
jnthn lizmat: Ah, right, we discussed that part, but never quite got around to it. :) 20:29
20:57 squashable6 left, squashable6 joined 21:35 kawaii left 21:36 kawaii joined 21:38 AlexDaniel left 22:02 Altai-man_ joined 22:05 sena_kun left 22:48 leont left
lizmat just ran a profile with %h{*}:delete, and only 6% of CPU was spent actually deleting the keys, and about 80% in generating the return values 22:52
which, in my benchmark, were just thrown away
23:01 lizmat left
timotimo send spesh logs :) 23:02
23:04 lizmat joined 23:55 squashable6 left 23:57 squashable6 joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!