This channel is intended for people just starting with the Raku Programming Language (raku.org). Logs are available at irclogs.raku.org/raku-beginner/live.html
Set by lizmat on 8 June 2022.
04:45 Some-body_ joined 04:46 DarthGandalf left 04:48 Some-body_ is now known as DarthGandalf 08:18 dakkar joined 13:13 jaguart joined 13:20 jaguart left 13:21 jaguart joined 14:08 kjp left 15:12 kjp joined 17:00 dakkar left
_grenzo I keep segfaulting raku 20:15
$script-path.IO.lines.rotor($batch-size, :partial).race.map: -> @batch { my $q = @batch.join("\n"); try { my @res = $dbh.execute($q).values; } CATCH { default { say "Error: $_"; } } $dbh.finish; $lines-done += @batch.elems; };
batch-size = 1000, The file $script-path has 4_581_526 lines of sql updates 20:16
Welcome to Rakudo™ v2024.05. Implementing the Raku® Programming Language v6.d. Built on MoarVM version 2024.05. 20:17
antononcube I detect very recent migration from moar-2024.04 to moar-2024.05. 20:18
(Maybe with hopes for less segfaulting...)
_grenzo I have both...they both segfault 20:19
antononcube 🤣 2️⃣
_grenzo You know of any way to gather info on why they segfault 20:20
antononcube Hmm... maybe the professional Comma IDE versions help with that... 20:23
_grenzo I'm guesing it has something to do with the file size as it can finish other smaller files 20:31
gfldex _grenzo: if a Lock around `$dbh.execute($q).values` makes the segfault go away, then there is somthing in DBIish not quite threadsafe. 20:53
tonyo iirc DBIish is not threadsafe
_grenzo Using DB::Pg
but I'll check 20:54
Thanks
ab5tract I also wonder if there could be an issue with tallying the $lines-done 20:55
_grenzo There was originally as I was doing $script-path.lines.elems for a 4m+ file. It would die on that 20:56
I changed it to my $lines-total = qqx{wc -l $script-path}.words[0];
ab5tract Also, I believe that this map block will be executing in batches of 64 across N degrees 20:58
I think you might want to renove the rotor and adjust the batch size directly in race 20:59
At least I think that would make it easier to reason about
Re: $lines-total, did Raku segfault when you did IO.lines.elems? Or did you get an OOM? 21:01
_grenzo segfault 21:02
or sometimes just killed
ab5tract Dang :(
_grenzo lock did not seem to hellp 21:03
ab5tract What I meant about rallying $lines-done is that maybe multiple threads are trying to set that variable at the same time and cause a collision
s/rallying/tallying/ 21:04
_grenzo That could be
ab5tract Moving the $lock.protect to be around that line would test that theory 21:06
_grenzo Trying that now 21:07
that lock is looking goot so far 21:08
BTW removing rotor broke something 21:09
ab5tract Or also docs.raku.org/type/atomicint#sub_atomic-assign
Ah then keep it :)
Like I said, I don’t do much with race/hyper so I don’t have many tips on how to hold it correctly 21:10
_grenzo: please do report on any successful 4M+ SQL statement run :) (and of course any other issues you might encounter on the way) 21:24
_grenzo 1M worked.
I'll let you know on the others
trying the atomic-fetch-add on another 1M to see if it's faster than lock 21:26
ab5tract Cheers! The atomicint approach should be a bit faster than using a lock, but glad that it seems we fixed your segfault 21:30
librasteve ⚛️ 21:31
_grenzo Thank you! 21:38
Attempting 4.6M rows. Expect it to take 25min 22:05
Progress: 4579526/4581526 99.95634642% - 1504.612250511 22:30
25 min
4.6M rows successful. Thanks again

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!