🦋 Welcome to the MAIN() IRC channel of the Raku Programming Language (raku.org). Log available at irclogs.raku.org/raku/live.html . If you're a beginner, you can also check out the #raku-beginner channel!
Set by lizmat on 6 September 2022.
00:00 bdju left 00:01 bdju joined 00:11 Sgeo joined 00:14 nil78 left 00:17 nil78 joined 00:33 hudo left 00:34 hudo joined 00:41 hudo left 00:42 hudo joined, nativecallable6 left, nativecallable6 joined 00:55 yewscion joined 01:00 yewscion left 01:04 xkr47 left 01:10 xkr47 joined 01:25 jpn left 01:33 nil78 left 01:39 nil78 joined 01:56 simcop2387 left, simcop2387_ joined 01:57 simcop2387_ is now known as simcop2387 02:08 kylese left, kylese joined 02:27 nil78 left 02:29 nil78 joined 02:52 yewscion joined 02:56 yewscion left 03:08 jpn joined 03:12 jpn left 03:15 kylese left, kylese joined 03:35 zetaaaa left 03:45 yewscion joined 03:48 nil78 left 03:50 nil78 joined 03:51 yewscion left 04:47 nil78 left 04:59 nil78 joined 05:02 yewscion joined 05:06 yewscion left 06:00 Aedil joined 06:47 guifa left 06:57 maylay left 06:59 maylay joined 07:42 discord-raku-bot left, discord-raku-bot joined 08:01 nil78 left 08:05 nil78 joined 08:10 kaskal joined 08:11 pierrot joined 08:12 pierrot_ left, kaskal- left 08:42 Sgeo left 08:51 yewscion joined 08:56 jpn joined 08:58 yewscion left 09:13 dakkar joined
grondilu Is it reasonable to use *.raku as a marshalling method for a 250Mb or so database? 09:14
grondilu is trying to store 274,426 chess games as a move tree. 09:19
I mean it's going to be a huge tree so I'm not sure simply using *.raku is going to fly 09:23
after all I got a segfault when I tried converting it to JSON. 09:24
09:32 nil78 left 09:35 nil78 joined 09:52 nil78 left 09:57 apac joined 10:02 nil78 joined 10:03 Manifest0 joined
Manifest0 hi 10:03
after updating to the latest master, zef is failing with 10:04
"Cannot unbox a type object (Str) to a str"
when upgrading packages
*latest main 10:05
raku v2025.01-66-g045c280f9 10:06
paste.opensuse.org/pastes/d312b8d45018 10:20
lizmat Manifest0: are you on the "master" branch still? Not on "main" ? 10:29
grondilu: are you using the standard .raku logic on classes ? 10:30
grondilu: OOC, how are moves represented? 10:31
Manifest0 lizmat: i meant main. master cames from muscle memory, as in the company i work with, we use master 10:37
lizmat when you updated, did you just do a git pull and then make install ? 10:38
Manifest0 i use rakubrew
rakubrew build moar main
lizmat ah, ok, then maybe patrickb can help you: I don't use rakubrew 10:39
looking at the error, I would suggest trying to remove all precomp dirs (not sure whether that is applicable to rakubrew or not) 10:41
specifically try nuking ~/.raku/precomp
Manifest0 deleting ~/.raku/precomp didn't help 10:43
lizmat then I'd really refer to patrickb 10:50
fwiw, I have created a local dist with all the distributions that I need as dependencies, and every now and then I nuke rakudo's install dir and configure from scratch 10:51
and then run zef . --deps-only --/test in that dist to get all of the modules I need back
10:53 Aedil left
grondilu lizmat: I'm building a hash, and I marshall it into a file with the raku method. 10:53
lizmat: moves are sequences of strings.
lizmat and the keys are ? 10:54
grondilu it's a tree. so keys are the preceding moves.
for what it's worth the tree has just been done. It took 79 minutes. Now I'm trying to put it in a module.
initial data is this PGN file : database.nikonoel.fr/lichess_elite_2024-10.zip 10:56
patrickb Manifest0 Is the error also there when using 2025.01?
grondilu I could parse it with my Chess::PGN module 10:57
lizmat how much would you need of the headers ?
all?
have you tried .slurp.split("\n\n") -> $headers, $moves { 10:59
?
11:01 Aedil joined
grondilu parsing is not the issue, it worked fine. The problem is turning the lines into a tree. 11:09
I've just tried loading the module and the process was killed because of excess memory usage according to a GNOME notification. 11:10
Well, to be exact generating the tree worked. This is what took 79 minutes. But now I can't load it into a constant inside a module. 11:11
lizmat could you gist part of that tree to give me an idea ? 11:12
grondilu one moment 11:13
gist.github.com/grondilu/c313114f9...b90b74176a 11:15
I passed it through fmt -s |head
as it's obviously much larger than this.
lizmat indeed.. so that's hashes within hashes within hashes within hashes within hashes ... 11:16
grondilu that's what I meant by "tree"
a big one, admitedly 11:17
the .raku file is 190Mb
lizmat so the gist shows one game
grondilu no, just the first key 11:18
but yeah maybe it's one game
Manifest0 i think i solved my issue... I did: rakubrew switch system; rakubrew nuke moar-main; rakubrew build moar main; rakubrew build-zef
i'm installing now the modules 11:19
lizmat grondilu: so that I understand: that tree would be the key to all the games that ran that way, right ?
grondilu well, yes. All games starting with 1.Na3 are in the same hash. 11:20
I can make a short version with 100 games
lizmat no, I get what you're trying to do now :-) 11:21
feels to me the overhead of hashes is too large for this approach to work on current hardware 11:22
grondilu what else could I do then? 11:23
lizmat I'm trying some approaches.. 11:24
grondilu damn, even with just 100 games loading the module seems difficult.
lizmat looks like there are about 275K unique moves 11:26
(as in .words in the moves section) 11:27
grondilu that's rather about the number of games.
lizmat ah, yes
forgot a slip 11:28
ok, about 8800 unique moves 11:29
grondilu are you thinking of making an index for the moves? 11:30
lizmat yeah, 16bit ints per move should be enough
grondilu makes sense 11:31
lizmat basically encode each game as a 16bit array and have each game have an index value 11:32
grondilu I don't need to save the games individually, though. What I want is the tree structure. 11:33
but representing each move by an index would surely reduce the size of the tree, indeed. 11:34
lizmat it wouldn't much if you'd still use hashes, though 11:35
grondilu would lists take less spaces? 11:36
I mean there are not so many branches per node, so I probably don't really need hashes. 11:37
lizmat so that I get what you want to do with it: you want a fast way to find out all of the plays from a given game position, right ? 11:38
grondilu yes
but getting the position from a sequence of moves is very slow
also there is the issue with transpositions, so I thought it be best to get the list of previous moves as key, not the position. 11:39
so what I want is : from a given sequence of moves, which moves were played next and in which games? 11:40
hang on maybe I could just grep it. 11:41
lizmat ok, so sequence of moves, *not* the actual board state ?
grondilu yes, *not* the board state.
maybe one day I'll use that, but not for now.
lizmat ok, then I'd encode the games a 16-bit int arrays 11:42
grondilu and do a grep?
I'd be amazed if that were efficient enough, but maybe it is.
lizmat and if you'd want the games of a certain sequence., then first grep on the move index in the long list 11:43
and then grep on those only to see if they were the same sequence
the 16bit int arrays should be easily generated and compiled 11:44
and loaded
grondilu I suppose if I sort it all, the search should be fast.
lizmat if you don't need to refer to the actual game meta info, that'd be cool as well 11:45
and maybe it would make sense to not generate it as a 16bit int array, but as a big int
but then you would need to make the bigints as large as the largest game 11:48
which would be around a 2400 hex-digit number 11:50
grondilu I don't quite understand how I can have 32Gb of RAM and struggle to store a 256Mb database
lizmat each hash keeps the original key strings as 32-bit UTF-8, + the hashes version + overhead 11:51
the initial approach created a *lot* of hashes
and there could be some GC issues that you're tickling as well 11:52
grondilu so the first step is to make a list of all the unique moves in the database. 11:53
lizmat I did: .slurp.split("\n\n").map(-> $, $moves? { .words with $moves }).unique 11:54
grondilu then I encode each game into a list of move indexes, which I can turn into a big integer.
lizmat I would hold off on the big integer initially 11:55
build 16-bit index arrays, and create an array of that
grondilu ok, and then to search I do successive greps? 11:56
one grep for the first move, then an other for the second move and so on?
presumably if my array is sorted that could be fast. 11:57
lizmat then finding the same "path" to a board position, could be a single grep actually: inside the grep, check from the move number down to the first move, don't accept as soon as there is no match
going through 256K entries should be pretty fast if it's just index -> index lookups
grondilu I see. going backwards from last move to first, then? 11:58
lizmat from the move index you're interested in
the way I understand your question is: give me all the games that start with this move sequence 11:59
say 20 moves
grondilu that's exactly what I want indeed
lizmat you start down checking all of the games arrays on position 19 down to 0
grondilu gotcha 12:00
I'll work on it, thanks.
Manifest0 lizmat: Now it's getting stuck when performing the paraseq tests: github.com/lizmat/ParaSeq/blob/mai...kutest#L13 when the degree is 2
lizmat Manifest0: please ignore failures on ParaSeq :-( 12:01
Manifest0 ok
lizmat it's a known issue with ParaSeq, but haven't been able to determine whether it's ParaSeq's problem, or Rakudo :-( 12:02
Manifest0: OOC, are you on Intel hardware ? 12:03
in the sense of non-ARM
Manifest0 I'm on a server AMD EPYC processor 12:04
non arm
kernel linux 5.14 12:06
12:09 gabiruh left 12:16 grondilu left
lizmat ok, so it may be a JIT issue... as I can't reproduce the hang on my M1 MacOS box 12:18
12:21 gabiruh joined 12:27 guifa joined 12:32 jpn left
Manifest0 i'm also having another issue, but this time is installing OO::Monitors: github.com/rakudo/rakudo/issues/57...2653572174 12:39
antononcube @grondilu Have tried using CBOR instead of JSON/raku? 12:42
lizmat Manifest0: zef install issue confirmed
looking into it
Manifest0 :-) 12:43
antononcube BTW, for me, programming, writing, or presenting about Chess and Raku has the smallest audience-interest vs efforts-time ratio. 12:46
With Wolfram Language (WL) I got more interest or reader-traction, but that is expected, given WL's user-base. (Size and topics of interest.) 12:48
12:48 jpn joined
lizmat Manifest0: found the reason, was a change introduced after 2024.01 12:52
Manifest0: updated OO::Monitors now in the ecosystem 13:05
if you do a "zef update" and then try to install it, it might already work
yeah, worked for me 13:06
thanks for the report!
Xliff lizmat: What turned out to be the issue? His problem has the same and is filed under #5780 13:07
lizmat use of undocumented internal feature BUILDALL 13:09
Xliff OK. I'll see if that solves mine. Wait one.
lizmat the BUILDALL method doesn't anymore, it's called POPULATE now
*exists
13:18 yewscion joined
Xliff lizmat: Yep! That fixed the problem! Will close the issue. 13:21
lizmat++
lizmat no, please leave it open for now
Xliff Oops. Closed and I cannot reopen. 13:22
lizmat I can :-)
13:23 yewscion left
Xliff OK! 13:23
lizmat there shouldn't be an issue if you're calling BUILDALL, that's ok 13:25
it's just when you check for BUILDALL as a method inside a class itself, it won't be there anymore
m: class A { has $.a = 42 }; say A.^find_method("BUILDPLAN") 13:26
camelia (Mu)
lizmat m: class A { has $.a = 42 }; say A.^find_method("BUILDALL")
camelia BUILDALL
tbrowder just now noticed the update of OO::Monitors. i wonder if that fixed issu could have been what caused my raku core dumps. 13:34
anyhoo, i'm upgrading all now 13:35
lizmat possibly, but I doubt it, but would be good if it did :-) 13:36
13:42 Xliff left
Manifest0 OO::Monitors installed. Thanks lizmat! 13:43
lizmat I think I covered all ecosystem fallout now, I guess the next blin run might find some more 13:48
13:49 apac left 14:28 grondilu joined 14:36 grondilu left, grondilu joined 15:05 Sgeo joined 15:36 yewscion joined 15:58 apac joined 16:03 peder_ joined 16:11 peder left, peder_ is now known as peder 16:15 peder_ joined 16:21 yewscion left 17:09 jpn left 17:10 apac left 17:39 dakkar left 18:56 Aedil left 18:59 jpn joined 19:14 Aedil joined 19:16 ACfromTX joined 19:18 nil78 left 19:22 nil78 joined 19:40 dmvrtx_ left 19:42 dmvrtx joined 20:03 sena_kun joined 20:53 apac joined 21:04 jpn left 21:08 Aedil left 21:10 yewscion joined 21:11 jpn joined 21:15 yewscion left 21:26 jpn left 21:36 yewscion joined 21:37 jpn joined 21:41 yewscion left
tonyo ? 21:44
21:56 apac left
[Coke] want me to kick off a run? 22:11
it's been a few weeks, will do so 22:12
SmokeMachine Hi! a few time ago I talked here about an idea I had for by a single class have aggregation, commands and events. Early I realized that was not a good idea. Then I decided to go in a different way. I started writing Sourcing (github.com/FCO/Sourcing/tree/main/.../Account). It has a few different constructs: Events (what "stores" data), Projections (that consumes events and updates themselves and have "query" methods 22:27
(marked as "is query") that are atomic can/should return data), Aggregations (that are specialization of Projections that can also have "is command" methods that are also atomic and before running, it makes sure the most recent event was already processed to have the data done to validate the entry passed and then emit events representing what that command did). And Processes (process manager (Saga)) (That are specializations of
Aggregation that can call commands on multiple aggregations and have method to auto call compensation commands in case of error and auto run commands in case of timeout). On the example on the link, there is a Aggregation called Account with commands (and event handlers) to withdraw and receive amounts. Projections (AccountTotalReceived and AccountTotalSent). And a Process to create transactions to remove from one account and add
to another one, and rollback if something goes wrong... I'm still on the very beginning of the development. I'd love if someone would like to take a look and gimme some opinions...
22:37 yewscion joined 22:41 yewscion left 22:42 ryoskzypu left
SmokeMachine I just added a README to the repo LLM helped me to create: github.com/FCO/Sourcing/tree/main 23:14
23:14 ryoskzypu joined 23:26 jpn left 23:42 zetaaaa joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!