|
01:07
melezhik_ joined
01:10
melezhik_ left
04:41
melezhik_ joined
04:43
melezhik_ left
05:29
melezhik joined
05:46
melezhik_ joined
05:48
melezhik_ left
|
|||
| melezhik | [Coke]: the scenario is that . Say we have two modules in test list : A, B. And A depends on B. And A is tested first, when we say “test” we mean simple “zef install A” command that will 1. Install all module dependencies recursively ( zef install also carries out zef test command under the hood). As a result when A has been tested , B is tested as well. So when time comes to test B, zef will skip it by emitting “B | 05:52 | |
| is already installed” | |||
|
05:56
melezhik_ joined
|
|||
| melezhik | There is also “agents active sharing” almost implanted, when if module A is already tested on agent1, and in parallel scheduled for testing on agent2 ( which may happen ), orchestrator tries to notify agent2 that there is no need to test module A. | 05:57 | |
| Combining these two approaches , plus cache building mentioned before, I hope we are good to beat this 10 hours for 2000 modules threshold | 05:58 | ||
| Implanted -> implemented | |||
|
06:01
melezhik_ left
06:13
melezhik_ joined
06:18
melezhik_ left,
melezhik_ joined
06:27
melezhik_ left,
melezhik_ joined
06:35
melezhik_ left,
melezhik_ joined
06:45
melezhik_ left,
melezhik_ joined
06:50
melezhik_ left,
melezhik_ joined
07:01
melezhik_ left
07:18
melezhik_ joined
07:28
melezhik_ left,
melezhik_ joined
07:36
melezhik_ left,
melezhik_ joined
07:45
melezhik_ left,
melezhik_ joined
07:53
melezhik_ left
07:59
melezhik left
08:01
melezhik joined
|
|||
| ab5tract | last night I had a dream about secret agents with hidden aliases | 08:49 | |
| I blame this on the fact that I'm trying to figure out how to get ::T to consistently resolve to it's "real" type when ::T has been parameterized into a role | 08:50 | ||
| wlel how did we do last night? | 08:55 | ||
| melezhik: ^^ | |||
| melezhik | ))) | 09:12 | |
| 1000 modules round successful finished | |||
| It took less then 6 hours to complete 800 modules which is ok, but bear in mind that 1. Coke-bot join the party a bit late , mine agent is tiny , so halve of this time I guess we effectively had only your agent 2. Agent sharing was not implemented at this point , so I expect significant speed increase with new version of agents 3. We have not had any 599 or other errors and all agents worked correctly | 09:16 | ||
| ab5tract | what does agent sharing mean here? | 09:21 | |
| melezhik | “8:57 AM <melezhik> There is also “agents active sharing” almost implanted, when if module A is already tested on agent1, and in parallel scheduled for testing on agent2 ( which may happen ), orchestrator tries to notify agent2 that there is no need to test module A.” | 09:28 | |
| ab5tract | yes, this would be great | ||
| I would go one step further and submit all dependent modules as already tested | 09:29 | ||
| melezhik | I knew about that before just did not have time to implement. Please upgrade agent to the latest commit | ||
| ab5tract | switching from zef to using the zef Raku modules would make finding out what dependencies were successfully installed a trivial feature | 09:30 | |
| patrickb | I might want to join the party. I'd want to put it all in a qemu VM though. What OS would be best? Debian? | 09:56 | |
| melezhik | Cool 😎 | 09:58 | |
| Any Linux will work | |||
| github.com/melezhik/brownie/blob/m...DME.md#faq if you run under none x86-64 | 10:00 | ||
| [Coke]: ab5tract patrickb please rebuild agent from the lastest commit and run agents | 10:01 | ||
| This will ensure agents active sharing | 10:03 | ||
| “12:30 PM <ab5tract> switching from zef to using the zef Raku modules would make finding out what dependencies were successfully installed a trivial feature” yeah we can think about it in the next versions | |||
| ab5tract | restarting now | 10:04 | |
| melezhik | I run another 1000 modules round | 10:44 | |
|
11:10
melezhik_ joined
11:15
melezhik_ left,
melezhik_ joined
11:21
melezhik_ left,
melezhik_ joined
11:26
melezhik_ left,
melezhik_ joined
|
|||
| melezhik_ | . | 11:27 | |
| ab5tract please rebuild agent again from the latest commit, funny performance bug crept in, thanks | 11:28 | ||
| ab5tract | ok | 11:29 | |
|
11:32
melezhik_ left
|
|||
| lizmat | bisectable6: old=2023.01 dd 4.5 mod 2.5 | 12:25 | |
| bisectable6 | lizmat, Cannot find revision “2023.01” (did you mean “2025.01”?) | ||
| lizmat | bisectable6: old=2023.02 dd 4.5 mod 2.5 | ||
| bisectable6 | lizmat, Bisecting by exit code (old=2023.02 new=3e9f252). Old exit code: 1 | ||
| timo | no dd back then? | ||
| lizmat | no 2023.01 release ? | 12:26 | |
| timo | i'm guessing it was skipped, yeah | ||
| lizmat | ah, no, infix mod only accepting Ints I think | ||
| bisectable6 | lizmat, bisect log: gist.github.com/0a81e39ead9f2e590a...a3cfa6cee6 | 12:28 | |
| lizmat, (2024-02-04) github.com/rakudo/rakudo/commit/71...7429f88fb7 | |||
|
12:59
melezhik left
|
|||
| [Coke] | rebuilt, rerun | 13:04 | |
| happened to see a missing c library error - will find the list of things we pre-installed on blin to catch ads many of these as possible | 13:12 | ||
| Geth | Blin: d813e8a67a | (Will Coleda)++ | README.md Remove time estimate from 2021 |
13:14 | |
| [Coke] | bah, that list might be on my blin box itself and not checked in anywhere. | 13:16 | |
|
14:02
melezhik_ joined
|
|||
| melezhik_ | . | 14:02 | |
| ab5tract: are you in? don't see your agent ... | 14:04 | ||
| oh, see it now | 14:06 | ||
|
14:14
melezhik_ left
14:20
melezhik joined,
melezhik_ joined
|
|||
| melezhik_ | this is how agent sharing look like from job logs prospective - gist.github.com/melezhik/902306fb0...ceb668cec6 | 14:22 | |
| docker exec -it agent bash -c 'grep -RH "is already processed" ~/.sparky/projects/.reports/' | |||
|
14:27
melezhik_ left
14:40
melezhik_ joined
14:44
melezhik_ left
14:52
melezhik_ joined
|
|||
| patrickb | melezhik: what's a good agent count for 28 cores and 38g ram? | 15:16 | |
| *thread count | 15:17 | ||
| melezhik_ | technically threads count should be around number of cores | 15:22 | |
| patrickb | But then I run into oom | 15:26 | |
| maybe ram / 2 ? | 15:27 | ||
|
15:29
melezhik_ left
15:32
melezhik_ joined
15:38
melezhik_ left
15:50
melezhik_ joined
|
|||
| melezhik_ | ping from agent: stoic-mclean-01820586, version: 0.0.26, jobs-run-cnt: 19, max-threads: 19 | 15:51 | |
| oh, bug guns here )) | |||
| big guns | |||
| patrickb | ram / 2 is pretty tight. It almost starts to swap | 15:54 | |
| melezhik_ | so it's eating RAM, correct ? | 15:56 | |
| how much RAM did you allocate ? | |||
|
16:00
melezhik_ left
|
|||
| patrickb | the VM had 40 gb | 16:02 | |
| with 19 threads it nearly scratched the limit then started to swap again after a while | 16:03 | ||
| now I try 50gb / 19 threads. Let's see how that goes. | |||
|
16:04
melezhik_ joined
16:08
melezhik_ left
16:19
melezhik_ joined
16:22
melezhik_ left
16:29
librasteve_ joined
16:35
melezhik_ joined
|
|||
| melezhik | Ok. Now I have some fear o10r is down and probably because it has failed to handle feedback from so many agents / threads . Probably I need to do some throttle on its side . Sigh . I will let you know once I fix that … | 17:04 | |
|
17:18
linkable6 left,
notable6 left,
sourceable6 left,
greppable6 left,
committable6 left,
quotable6 left,
unicodable6 left,
evalable6 left,
tellable6 left,
coverable6 left,
bloatable6 left,
benchable6 left
17:22
[Coke] left,
quotable6 joined,
unicodable6 joined,
[Coke] joined,
committable6 joined,
evalable6 joined,
greppable6 joined,
sourceable6 joined,
notable6 joined,
tellable6 joined,
coverable6 joined,
bloatable6 joined,
benchable6 joined
17:25
linkable6 joined
|
|||
| melezhik | [Coke]: patrickb ab5tract I am fixing sparky main server after it crashed under overload . I am going to send a patch to agent code so that requests from agents to o10r will be throttled and don’t overload it . Sorry for inconvenience | 17:31 | |
| I will let you know once it’s settled | 17:32 | ||
| melezhik_ | github.com/melezhik/brownie/commit...afb0b306b0 | 18:13 | |
| fix for 010r overload | |||
| I guess this was patrickb: agent that killed 010r that send response that resulted in 19 parallel Rakudo processes on 010r side, not it's limited to 4 at a time | 18:16 | ||
| github.com/melezhik/brownie/blob/1...wfile#L232 | 18:17 | ||
|
18:21
melezhik_ left
18:23
melezhik_ joined
|
|||
| patrickb | restarted | 18:29 | |
| [Coke] | restarted | 18:35 | |
|
18:47
MasterDuke joined
|
|||
| MasterDuke | is t/04-nativecall/26-varargs.t expected to fail test 4 ('Can pass pointers to typed ints') right now? | 18:48 | |
|
18:58
finanalyst joined
|
|||
| melezhik_ | [Coke]: patrickb: I am still testing , it will need little time, I will let you when to upgrade agent and restart it, thanks for your patience and help ) | 19:02 | |
| cc ^^ ab5tract: ) | |||
| ab5tract: [Coke]: I've added btw this feature - when installed dependencies for any module marked as already processed , this should save a lot of time, not asking to test modules that are already covered through dependence chain ... | 19:04 | ||
| this is an example for `zef install AlgorithmsIT` - gist.github.com/melezhik/d83d69460...6494524edd | 19:06 | ||
| sorry wrong link | |||
| [Coke] | you need not just ver, but also auth and api if present, btw. | 19:17 | |
| Otherwise you end up with the JSON::Class issue. | |||
| melezhik_ | so yeh - here is an example gist.github.com/melezhik/4ae510b30...685ea541b5 | ||
| [Coke] | (you *cannot* assume that a higher version number meets your requirement) | ||
| melezhik_ | `zef install Apache::LogFormat` also installs `DateTime::Format:ver<0.1.5>:auth<zef:raku-community-modules>:api<1.0>` as deps | 19:18 | |
| so task.check parser - github.com/melezhik/brownie/blob/8...k.check#L5 extracts only module name part, without version and mark it as processed | 19:19 | ||
| patrickb | MasterDuke: no, that test is not expected to fail | ||
| MasterDuke | patrickb: hmm, ok, thanks | 19:20 | |
| patrickb | libffi only? | ||
| melezhik_ | [Coke]: the idea is to parse `zef install Foo` output and collect all `\s* "Installing:" \s+ (\S+?) ":ver" \S` lines as dependencies | ||
| so from zef output we know for sure they are installed, so we can safely mark them as installed | 19:21 | ||
| MasterDuke | failed on my branch, but now i'm seeing the fail on main also. but i guess some workspace cleanup hasn't been thorough enough... | ||
| melezhik_ | the only issue may be here if someone freeze dependencies in META6.json | ||
| so zef would install not latest ( that works for this case ) but some frozen/old dependency, but I'd suggest to ignore that, as promotion of such case per whole module list might be very little , what you think ? | 19:23 | ||
| "(you *cannot* assume that a higher version number meets your requirement)" this is just (hopefully ) very good approximation, in the future we can come up with something better, however the problem is that input list of modules for tests comes without versions, at least with current logic | 19:24 | ||
| MasterDuke | hm. the test compiles some stuff. i wonder what is precomped and how to clean those out... | 19:25 | |
| melezhik_ | so I assume this another example should be good enough - deps installed during installation of module App::AizuOnlineJudge: $["OpenSSL", "IO::Socket::Async::SSL", "OO::Monitors", "IO::Path::ChildSecure", "Base64", "HTTP::HPACK", "Cro::Core", "Cro::TLS", "if", "Crypt::Random", "MIME::Base64", "Digest", "Digest::HMAC", "JSON::JWT", "DateTime::Parse", "TinyFloats", "CBOR::Simple", "Log::Timeline | ||
| etc | |||
| MasterDuke | patrickb: when recompiling: | 19:27 | |
| ``` | |||
| t/04-nativecall/26-varargs.c: In function ‘va3’: | |||
| t/04-nativecall/26-varargs.c:64:13: warning: comparison is always false due to limited range of data type [-Wtype-limits] | |||
| 64 | *a1 == -1 && | |||
| | ^~ | |||
| ``` | |||
| probably unrelated | 19:28 | ||
| afk for a bit | 19:29 | ||
|
19:33
MasterDuke left
19:39
melezhik left
|
|||
| melezhik_ | [Coke]: patrickb: ab5tract: please upgrade agent from the latest commit. And restart it. Thanks. I've done several important improvements, including making installed dependencies as processed to increase performance | 19:48 | |
| in the future , starting from this upcoming version, you may upgrade agents, without restarting containers ( this will allow keep already built cache ) - github.com/melezhik/brownie/blob/m...tarting-it | 20:03 | ||
|
20:06
melezhik_ left
20:07
melezhik_ joined
|
|||
| [Coke] | restarted | 20:08 | |
| melezhik_ | [Coke]++ | 20:12 | |
| run 2000 module round, patrickb: agent has a bit old version, but I'd like to see the timing anyway , [Coke]: agent is uptodate, ab5tract: whenever you are ready, please upgrade your agent from last commit and restart it, thanks) | 20:26 | ||
| we've done 100 tests in 8 minute - impressive , even with suboptimal patrickb: agent ! too fast to believe )) | 20:33 | ||
|
20:52
melezhik_ left
|
|||
| patrickb | MasterDuke: Just did another quick test run locally. All green. | 21:38 | |
| tellable6 | patrickb, I'll pass your message to MasterDuke | ||
| patrickb | I do now realize though that the vararg tests are pretty dependant on the type sizes. So I kind of expect the tests to fail depending on compiler / os / arch. | 21:39 | |
| I don't immediately know how to best fix this... Ideas? | |||
|
21:48
El_Che left
21:54
lizmat_ joined
21:58
lizmat left
23:02
melezhik_ joined
23:03
lizmat_ left,
lizmat joined
23:07
melezhik_ left
23:10
Geth left,
Geth joined
23:52
melezhik_ joined
23:54
melezhik_ left
|
|||