|
Welcome to the main channel on the development of MoarVM, a virtual machine for NQP and Rakudo (moarvm.org). This channel is being logged for historical purposes. Set by lizmat on 24 May 2021. |
|||
| MasterDuke | timo: can you think of a benchmark where adjusting the nursery might be beneficial? | 00:56 | |
| it all comes back to nfas | 02:34 | ||
| i added an MVM_dump_backtrace(tc) call to set_size_internal() (which created a 96gb log file when building CORE.c) to see where they were all coming from | 02:36 | ||
| 264002 at NQP::src/vm/moar/QAST/QASTCompilerMAST.nqp:2304 (/home/dan/Source/perl6/install/share/nqp/lib/QAST.moarvm:get_callsite_id_from_args) | |||
| 271489 at NQP::src/QAST/Op.nqp:7 (/home/dan/Source/perl6/install/share/nqp/lib/QASTNode.moarvm:new) | |||
| 350887 at <unknown>:1 (/home/dan/Source/perl6/rakudo/blib/Perl6/Grammar.moarvm:stdstopper) | |||
| 405184 at src/Perl6/Optimizer.nqp:2289 (/home/dan/Source/perl6/rakudo/blib/Perl6/Optimizer.moarvm:do) | |||
| 414458 at src/Perl6/Grammar.nqp:3751 (/home/dan/Source/perl6/rakudo/blib/Perl6/Grammar.moarvm:termish) | |||
| 577503 at NQP::src/vm/moar/QAST/QASTOperationsMAST.nqp:108 (/home/dan/Source/perl6/install/share/nqp/lib/QAST.moarvm:compile_mastop) | |||
| 603694 at NQP::src/QRegex/Cursor.nqp:729 (/home/dan/Source/perl6/install/share/nqp/lib/QRegex.moarvm:!protoregex) | |||
| 825573 at <unknown>:1 (/home/dan/Source/perl6/rakudo/blib/Perl6/Grammar.moarvm:_ws) | |||
| 1439631 at NQP::src/QRegex/NFA.nqp:1044 (/home/dan/Source/perl6/install/share/nqp/lib/QRegex.moarvm:run_alt) | |||
| 1719069 at NQP::src/QRegex/Cursor.nqp:559 (/home/dan/Source/perl6/install/share/nqp/lib/QRegex.moarvm:!cursor_capture) | |||
|
07:55
sena_kun joined
08:10
sena_kun left
|
|||
| timo | that's every set_size_internal, and not the ones that actually cause a realloc or malloc? | 16:10 | |
|
16:21
japhb_ left
17:09
sena_kun joined
17:27
sena_kun left
17:28
sena_kun joined
|
|||
| timo | github.com/facebook/zstd/blob/dev/..._format.md :eyes emoji: | 18:22 | |
| lizmat | any application you're considering? | 18:31 | |
| timo | spesh logs can become pretty lorge | 18:42 | |
| but it's still very useful to be able to skip around the contents very fast when looking for stuff | 18:43 | ||
| if the format we spit out isn't easily readable with just the zstd commandline tool, though, that's not attractive. we have a specific format for heap dumps where individual chunks are zstd compressed, but it's not useful to look at it with "less" or so, so there i don't see it as much of a problem | 18:45 | ||
| but you often like to grep through a regular spesh log, for instance | |||
| it sounds like having a trained dictionary before compression also improves compression speed, so it might be cool to have a zstd dictionary for spesh logs if i add zstd support for spesh logs in the first place | 18:47 | ||
| there's a proposed "dictionary inside zstd file" format, but the pull request has been dormant for years, even though it kind of seems like it's ready to go? | 18:48 | ||
| otherwise a cheeky `moar --get-zstd-dict foobar.dictfile` could be okay, since the dictionary has to be available inside of moar already for use in compression | 18:50 | ||
| 15315 7.4G -rw-r--r--. 1 timo timo 7.4G Aug 14 23:26 /tmp/moar-1212606.speshlog.txt | 18:51 | ||
| 15330 8.2G -rw-r--r--. 1 timo timo 8.2G Aug 14 23:33 /tmp/moar-1221195.speshlog.txt | |||
| oops that's even a little more than i expected | |||
| maybe zstd's "independent chunks" mode is good enough to be somewhat-seekable. i'm benchmarking a whole bunch of different parameters with dictionary and without etc etc for compression ratio and read/write speed | 19:22 | ||
| even without using a dictionary, and even with chunking the file into a fresh, unrelated chunk every 2 MB of input data, even with the lowest compresion ratio / highest compression speed, the size reduction is about 13x | 19:25 | ||
|
20:05
MasterDuke left
|
|||
| timo | Frames Skips Compressed Uncompressed Ratio Check Filename | 20:51 | |
| 556632 0 184 MiB 1.47 GiB 8.176 None /tmp/moar-3008468.speshlog.txt.zst | |||
| this is with writing lots of small bits in the middle, sometimes individual lines, as "full" zstd frames, so wasting a lot of potential ratio in order to have simpler code | 20:52 | ||
| [Coke] | jj | 22:02 | |
| ww | |||
|
22:03
sena_kun left
|
|||