[01:08] *** maggotbrain joined [02:20] ¦ nqp-configure: vrurg self-assigned Cannot determine the brand of your nmake utility https://github.com/Raku/nqp-configure/issues/19 [06:57] *** hungryd12 joined [07:00] *** hungrydonkey left [07:33] *** lichtkind joined [08:01] *** hungrydonkey joined [08:03] *** hungryd12 left [08:04] Altai-man: just noticed that 2020.02.1 release announcement talks a bit too much about Perl 6 [08:04] AlexDaniel, I'll pass your message to Altai-man_ [08:10] *** hungryd57 joined [08:12] *** hungryd57 left [08:13] *** hungrydonkey left [08:13] *** hungrydonkey joined [08:14] *** AlexDaniel left [08:27] *** AlexDaniel joined [08:27] *** AlexDaniel left [08:27] *** AlexDaniel joined [08:32] *** Kaeipi left [08:33] *** Kaiepi joined [08:46] Files=1306, Tests=111242, 215 wallclock secs (29.11 usr 8.47 sys + 3017.92 cusr 278.84 csys = 3334.34 CPU) [09:26] *** squashable6 left [09:27] *** squashable6 joined [09:30] *** sena_kun joined [09:35] MasterDuke: funny piece of code, that, using a native str as an index in a list [09:35] also, "if nqp::ord($name) == 36 && ($name eq '$!from' || $name eq '$!to') {" feels like a premature optimization nowadays ? [09:41] well, the native str would be one of the ascii digits at that point `if nqp::ord($name) < 58`, but yeah [09:41] m: use nqp; my int $a; my str $name = q/$bar/; for ^1000000 { ++$a if nqp::ord($name) == 36 && ($name eq q/$foo/ || $name eq q/$bar/) }; say $a; say now - INIT now [09:41] rakudo-moar 5610416c8: OUTPUT: «1000000␤0.2328733␤» [09:41] m: use nqp; my int $a; my str $name = q/$bar/; for ^1000000 { ++$a if $name eq q/$foo/ || $name eq q/$bar/ }; say $a; say now - INIT now [09:41] rakudo-moar 5610416c8: OUTPUT: «1000000␤0.08986747␤» [09:41] that's about 2.5x faster [09:42] I don't know how hot that loop is, but that would appear to be an easy win [09:42] that whole MATCH function is expensive and called a lot when parsing, so could be good [09:43] I think that piece of code is a leftover from the Parrot days [09:43] when string matching was relatively expensive, and ord was relatively cheap [09:43] now that we have proper grapheme support, I think it's the other way around now [09:46] MasterDuke: also another win: I see several places checking for CCLASS_ALPHANUMERIC and then checking for underscore: nqp::ord(95) [09:46] there's actually a CCLASS_WORD nowadays that combines them [09:47] so the extra check for nqp::ord(95) can then be removed [09:57] *** ggoebel joined [10:02] nice [10:05] .tell nine we may need to treat compiled scripts differently that precompiled modules because the end goal of the gsoc project was deliverable was to "Add a --compile option to perl6 which will generate a foo executable PE or ELF formatted binary for a given foo.pl6 user program which will facilitate self-contained deployment and/or deployment dependent upon the system shared version of Perl 6 which is installed." [10:05] ggoebel, I'll pass your message to nine [10:07] ggoebel: I don't read anything in that which would warrant different treatment. After all the self-contained program will want to use modules as well [10:12] the project summary references the .net single file distribution... but the link is broken. I think I've found it here: https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md [10:14] If I were to implement self-contained executables, I'd implement a new CompUnit::Repository and a new CompUnit::PrecompilationStore and fix the few remaining places in CompUnit::PrecompilationRepository where we still assume that we're dealing with files. [10:15] that was the fuzzy direction I thought things might need to go. but not being intimately familiar with everything, I was hesitant to suggest it. [10:17] where the CU:PrecompilationStore would allow you to bundle the module dependencies within the self-contained executable and load them directly into memory from there [10:17] here is the link to the gsoc project summary: https://yakshavingcream.blogspot.com/2019/08/summer-in-review.html [10:18] *** lichtkind_ joined [10:20] *** lichtkind left [10:20] the self-contained program may at one extreme end of the spectrum want to bundle _everything_ into it. So that you have a stand-alone relocatable binary with no dependency on raku being installed on the system [10:25] Precompiled scripts sound rather orthogonal to self-contained programs to me anyway [10:27] yes Perl5's PAR provides self-contained executables without precompilation [10:31] As do you apparently :) "And with that, I had a way to embed Perl 6 source code into an ELF file, and run it! It was time to move onto the next steps, figuring out how to run MBC directly, and adding the --compile flag to perl6." [10:32] I imagine you might be able to load the precompiled bytecode in a self-contained program without performing whatever tests are required to decide when to recompile things? [10:34] FWIW: I am related to, but am not pamplemouse (the gsoc implementer) [10:34] A couple years ago I realized that we're actually already at a point very close to being able to ship binary only distributions. In a normal program run we never touch the installed modules' source files. [10:34] Ah, that clears that confusion :) [10:34] I thought you have just changed nick or so [10:35] I encouraged them to get involved with raku via the gsoc project, but it is their work not mine. [10:36] is there much overhead in deciding when things need to be re-precompiled? I've never really understood how that works. [10:37] The overhead is reasonably small. It's all based on "trust the SHA". [10:38] Once you understand the problem, the solution should be rather straight forward to understand. [10:38] so a checksum for each precompiled module? [10:39] I guess I should go read the code... [10:39] To decide whether a precomp file is up to date, we need to know whether it still reflects the sources or not and whether its dependencies are in the exact same state as when we precompiled [10:40] Therefore precomp files can become invalid when you change the source (of course) or when the result of resolving the dependency specifications would change from the time we precompiled to the time we loaded. [10:41] The latter may change simply because one installed a newer version of a dependency (the classic case). It can also change if a new repository was added to the $*REPO chain and this repository contains a different version that would be preferred. I.e. raku -I/some/dir [10:42] So to a first approximation, a precomp file is safe if all repositories have the exact same state as when we precompiled. I.e. nothing got installed or uninstalled and the sources are identical. This is what I implemented first. [10:43] Does this mean we perform a checksum on every source file before using the associated precompiled bytecode? I appreciate you taking the time to answer my questions, but I don't want to take your time if there is a design doc or I should be figuring it out for myself by reading the code. [10:43] For this each repository generates an identifier (id) which reflects the full contents of the repository. Think a SHA hash of all installed modules. We record this hash in the precomp file and can compare when loading [10:43] Well, this may actually become the new "general overview" documentation :) [10:44] This first implementation works, but of course it has a major drawback: whenever we install any module, all precomp files will be invalidated. So we need to be a bit smarter. [10:45] When we detect that there was a change in the repository chain's identity - either by a repository getting added or removed or by a change in its contents, i.e. a module getting installed - we re-resolve the precomp file's dependencies. [10:47] so when the repo id SHA checksum fails, all the precomp files get rewritten. I think I'm with you. [10:47] So we take the dependency specification as given in the use statement (use Foo:auth/me/:ver(v1.*)) and check what precomp file we'd end up with. If it's the same, everything is fine. To do this we actually record this CompUnit::DependencySpecification in the precomp file's header. [10:48] As a side node: this re-rechecking of dependencies can actually be turned off via the RAKUDO_RERESOLVE_DEPENDENCIES=0 environment variable as a workaround for BEGIN time EVAL issues. That's because we EVAL the stringified CompUnit::DependencySpecification to be able to work with it. [10:49] lizmat: i made the $name eq change, but changing the 3 instances of CCLASS_ALPHANUMERIC + nqp::ord(95) to just CCLASS_WORD caused the nqp compile to die with `Use of undeclared variable '$diff-1' at line 321, near ") ;\n " at gen/moar/stage1/NQPHLL.nqp:1028 ...` [10:49] hmmm... that's weird [10:49] There are situations where the user knows that the dependencies are the same, like when packaging modules for Linux distros. In this case the dependencies are handled by a trustworthy mechanism outside rakudo. [10:49] pretty sure i didn't have a typo, but i'm going to double check now [10:50] then either the documentation is wrong, or something else is not properly understood [10:51] nine: thank you for taking the time to explain. and thank you for your work on raku [10:52] There's another optimization we use that has visible side effects: module distributions are specified as immutable, i.e. Foo:ver<0.1> must always contain the exact same sources. The module loader relies on this. This means that we don't actually have to look at the source files at all. Instead we just take a SHA of the distribution's meta data. We could even get by with just the distro's long name [10:54] Oh, we actually do. The SHA identifying a distribution is generated by Distribution.id which only looks at name, auth, ver and api [10:55] lizmat: well, the docs have CCLASS_WORD also including 'Nd' [10:56] *** Altai-man_ joined [10:56] MasterDuke: so does ALPHANUMERIC [10:56] only ALPHABETIC doesn't have that [10:56] I was hoping there would be performance gains to be had from self-contained programs that didn't have to perform checksums or file io before loading the precompiled bytecode. But it sounds like there might not be much. [10:56] oh, two of the places are ALPHABETIC, one is ALPHANUMERIC. i'll back out those two places [10:57] yeah, you should only do this optimization for ALPHANUMERIC, *not* ALPHABETIC [10:58] *** sena_kun left [10:58] yeah, now it's fine [10:59] I've literally spent hundreds of hours on getting the module loader as fast and well scaling as possible while still allowing distro packages to just put files into place whithout having to update some registry. [10:59] I seriously hope that I have not left any low hanging fruit. Though of course that would be fantastic :) [11:00] The upshot is that we're actually much faster than Perl and Python when loading large applications, as long as they are precompiled. [11:02] m: dd 1,2,3 ... * > 0 # this feels wrong [11:02] rakudo-moar 5610416c8: OUTPUT: «(1,).Seq␤» [11:03] Really what happens, when you load a precompiled module with a huge dependency tree is that we do a directory listing of all CompUnit::Repository::Installation's "dist" directories (no need to actually read those files) to generate the repo ids. We read all files in the appropriate "short" directory of those repositories. [11:03] m: dd 1,2,3 ... * > 5 # guess it's a consequence of this [11:03] rakudo-moar 5610416c8: OUTPUT: «(1, 2, 3, 4, 5, 6).Seq␤» [11:03] Then we already have the desired precomp file. This contains a list of exact file names of the whole dependency tree. So read the precomp file, get a full list of everything else you need to read + checksums of those files. [11:04] Then we read those dependencies which have their own checksum right there in the header, so we just need to read that line to be able to compare with our list of checksums. Then we continue to load the byte code. [11:05] I see a possibility for improving performance mostly be parallelizing this check of precomp files which may be faster once that list is in the 100s of modules (like in one of our applications) [11:05] Also MoarVM verifies bytecode when loading a module. This may be run in parallel as well. [11:11] That's only half true [11:11] It verifies the bytecode lazily, on the first time a frame is hit [11:11] Oh, yes, there's that [11:12] So if you load a module with 100 frames but only call 5 of them, you'll only verify those 5. [11:12] We could actually add Yet Another Thread (tm) which verifies in the background ;) [11:12] Hm, but is it a win? [11:13] iirc we also lazily build some data structures in memory at the time we verify that relate to the frame [11:13] It may be but my gut also says that it won't be worth it for a long while. That C code is pretty fast. Lots more important places to improve [11:13] So if we do it speculatively on frames we don't need, we blow up our memory use [11:13] Even before considering we mmap the bytecode so we can demand-page it [11:45] *** pamplemousse joined [11:48] *** lichtkind_ left [11:57] *** MasterDuke left [12:07] nine: Do you know where that directory file lives? I was on the path of to essentially duplicating the work for finding all used files, so finding and using that instead would be wonderful [12:16] pamplemousse: a CompUnit::PrecompilationUnit has a dependencies method that gives you the full list [12:16] lizmat: it's not wrong [12:17] lizmat: unless I'm missing something [12:17] *** MasterDuke joined [12:17] AlexDaniel:, yeah, just use ...^ if you don't want the last one [12:17] lizmat: … includes the last element that matched, …^ doesn't [12:17] nine: Nifty, thanks [12:19] I'm not sure we actually have a public API that will give you a PrecompilationUnit though... [12:19] maybe we should :-) [12:20] Well, the precompilation store's load-unit method returns one [12:42] ¦ nqp: e4db2f51f6 | (Daniel Green)++ | src/QRegex/Cursor.nqp [12:42] ¦ nqp: Some minor optimizations to NQP matching [12:42] ¦ nqp: [12:42] ¦ nqp: Nowadays checking string equality is faster than nqp::ord, so get rid of [12:42] ¦ nqp: an old optimization. [12:42] ¦ nqp: [12:42] ¦ nqp: Also, CCLASS_WORD is the same was CCLASS_ALPHANUMERIC + '_', so just use [12:42] ¦ nqp: that. [12:42] ¦ nqp: review: https://github.com/Raku/nqp/commit/e4db2f51f6 [12:45] MasterDuke: any visible improvements ? [12:45] no, but i wasn't doing really extensive benchmarking [12:51] *** ggoebel_ joined [12:54] *** ggoebel left [12:57] *** sena_kun joined [12:58] *** Altai-man_ left [13:07] *** pamplemousse left [13:09] *** pamplemousse joined [13:59] *** ggoebel__ joined [14:01] *** ggoebel_ left [14:06] *** robertle joined [14:28] *** ggoebel_ joined [14:30] *** ggoebel__ left [14:39] <[Tux]> Rakudo version 2020.02.1-357-g5610416c8 - MoarVM version 2020.02.1-153-g9bb7a1850 [14:39] <[Tux]> csv-test-xs-20 0.379 - 0.392 [14:39] <[Tux]> csv-ip5xs 0.946 - 0.949 [14:39] <[Tux]> test-t --race 1.016 - 1.032 [14:39] <[Tux]> test-t 2.826 - 2.938 [14:39] <[Tux]> test 8.278 - 8.620 [14:39] <[Tux]> csv-ip5xs-20 10.243 - 10.394 [14:39] <[Tux]> test-t-20 --race 13.982 - 14.084 [14:39] <[Tux]> csv-parser 26.668 - 27.582 [14:39] <[Tux]> test-t-20 49.179 - 49.798 [14:43] *** robertle left [14:56] *** Altai-man_ joined [14:58] *** sena_kun left [16:00] *** hungrydonkey left [16:03] *** ggoebel__ joined [16:06] *** ggoebel_ left [16:20] *** ggoebel joined [16:20] *** ggoebel__ left [16:50] *** cognomin_ joined [16:53] *** ggoebel left [16:54] *** cognominal left [16:57] *** sena_kun joined [16:59] *** Altai-man_ left [17:13] *** maggotbrain left [17:32] *** cognominal joined [17:36] *** cognomin_ left [17:51] *** ggoebel joined [18:02] *** ggoebel_ joined [18:04] *** ggoebel left [18:56] *** Altai-man_ joined [18:58] *** sena_kun left [19:36] i'm not really sure what TimToady meant here https://colabti.org/irclogger/irclogger_log/perl6-dev?date=2017-10-04#l14 "caching the first decision based on initial character would cut down the number of initial states we have to examine, for instance" [19:37] so i can't tell, is it still relevant? [19:49] I guess in the case of: [19:50] foo | bar | baz [19:50] caching the first character somewhere would only need to examine "foo" if the first is an "f", and "bar" and "baz" if the first is a "b", and none if something else ? [19:51] but wheyer [19:51] but whether that is still relevant? I'm not sure [19:51] not a lot has changed in that code for the past 5 years afaik [19:52] but perhaps current speshing / inlining / JITting has made these caches more of a burden than of a boon [20:04] *** MasterDuke left [20:12] *** MasterDuke joined [20:13] *** ggoebel__ joined [20:16] *** ggoebel_ left [20:48] m: dd ("c","c","c" ... *).head(10) [20:48] rakudo-moar 5610416c8: OUTPUT: «("c", "c", "c", "c", "c", "c", "c", "c", "c", "c").Seq␤» [20:48] m: dd ("c","c","c" ... "g").head(10) [20:48] rakudo-moar 5610416c8: OUTPUT: «("c", "c", "c", "d", "e", "f", "g").Seq␤» [20:48] m: dd (1,1,1 ... *).head(10) [20:48] rakudo-moar 5610416c8: OUTPUT: «(1, 1, 1, 1, 1, 1, 1, 1, 1, 1).Seq␤» [20:48] m: dd (1,1,1 ... 10).head(10) [20:48] rakudo-moar 5610416c8: OUTPUT: «().Seq␤» [20:48] these all feel very inconsistent [20:49] m: dd (1,1,1 ... 1).head(10) [20:49] rakudo-moar 5610416c8: OUTPUT: «(1,).Seq␤» [20:49] m: dd ("c","c","c" ... "c").head(10) [20:49] rakudo-moar 5610416c8: OUTPUT: «("c",).Seq␤» [20:50] at least these two last ones are consistent [20:56] *** pamplemousse left [20:57] *** sena_kun joined [20:58] *** Altai-man_ left [21:10] lizmat: which is why I say that we should define some easy to understand behavior first [21:11] instead of trying to DWIM in all cases in which case nobody has a clue what it actually does [21:11] sourceable6: .kv() [21:11] AlexDaniel, https://github.com/rakudo/rakudo/blob/5610416/src/core.c/List.pm6#L746 [22:01] *** sena_kun left