timo the moarvm debian package is now in transit between sid and trixie, and nqp and rakudo are also going to go into trixie now 00:10
release.debian.org/britney/update_output.txt - here you can see that moarvm, nqp, and rakudo are all in a "final:" line in a block with "SUCCESS"
release.debian.org/doc/britney/sho...tions.html this explains what britney does 00:11
coleman Nice 03:07
patrickb lizmat: I'm thinking through the "$path.spurt: $source.AST(:compunit).rakudoc.gist" part. If an attacker wants to corrupt the process they would add code in a begin block in the source to write out a separate executable file, and exec that. Then they can write any bytecode they want to $path. Evaling $path is then not safe. 07:26
That's why I see the need for a pure RakuDoc parser that is failing to parse anything but valid RakuDoc. 07:28
In your example $path is the intermediate format. We must assume a potential attacker has full control to put any content they want into that intermediate file. So the thing outside the sandbox that processes the intermediate file must not allow anything in the intermediate file to be executed. 07:32
I don't fully understand the implications, but it seems to me, that calling `.EVAL` on some input could potentially run code from that input.
timo [2024-09-12] moarvm 2022.12+dfsg-2 MIGRATED to testing (Debian testing watch) 08:01
patrickb timo++ 08:05
08:18 japhb left 08:40 sena_kun joined
Geth rakudo/windows_latest_azure_pipeline: ebd311e99b | (Timo Paulssen)++ | azure-pipelines.yml
get new moar telemetry additions
08:54
rakudo/windows_latest_azure_pipeline: f43de5f9e3 | (Timo Paulssen)++ | azure-pipelines.yml
new telemeh tweaks. try running without jit.
09:21
09:24 sena_kun left 09:26 sena_kun joined, sena_kun left 09:34 japhb joined 09:57 japhb left
Geth rakudo/windows_latest_azure_pipeline: c79992ea0a | (Timo Paulssen)++ | azure-pipelines.yml
try getting some windows event log output
09:59
09:59 japhb joined
Geth rakudo/windows_latest_azure_pipeline: ecb3d7255b | (Timo Paulssen)++ | azure-pipelines.yml
output some random windows events
10:48
rakudo/windows_latest_azure_pipeline: 737fffbcd6 | (Timo Paulssen)++ | azure-pipelines.yml
well, that was a dumb thing to miss
10:59
rakudo/windows_latest_azure_pipeline: c1a971006f | (Timo Paulssen)++ | azure-pipelines.yml
finish building rakudo with jit turned off

so we can have a cached moarvm hopefully.
11:13
11:33 finanalyst joined 11:37 finanalyst left
lizmat patrickb: understood 12:00
patrickb: did you already start on a SafeRakuDoc module ?
patrickb lizmat: No. I'm still in the brainstorming phase 12:02
Also I don't know if any changes to the RakuDoc spec would actually be needed. I know too little. I just learned that the new RakuDoc already differentiates ambient code. So maybe the spec is already fine. 12:04
lizmat "differentiates ambient code" what do you mean? 12:05
patrickb In github.com/Raku/RakuDoc-GAMMA/blob..._3.rakudoc the term "ambient context" shows up quite often. So the separation of what's rakudoc and what's not is already part of the spec. So maybe we don't need a SafeRakuDoc, but just a CleanRakuDoc that doesn't have any non-RakuDoc stuff mixed in. 12:11
Sadly the spec says: "A file with mainly RakuDoc source has a file extension B<.rakudoc>." and not "exclusively". 12:14
lizmat please make an issue for that :-) 12:15
patrickb My current, maybe false, undestanding is that we'd need to build two things: 1. A tool to strip all non-rakudoc from a file, leaving a clean RakuDoc file. 2. A pure RakuDoc parser that is unable to parse anything but clean RakuDoc.
lizmat the first is pretty easy for me to do 12:18
except that I seem to be running into a RakuAST issue with BEGIN blocks not being able to run in code that's compiled with .AST 12:19
patrickb github.com/Raku/RakuDoc-GAMMA/issues/48 12:20
There we go.
lizmat patrickb++ 12:21
Geth rakudo/windows_latest_azure_pipeline: 70a923f397 | (Timo Paulssen)++ | azure-pipelines.yml
get before/after event log counts, and grab some events

from semi-randomly picked categories
12:49
timo Downloaded 0.0 MB out of 115.9 MB (0%). 12:53
Downloaded 119.3 MB out of 115.9 MB (103%).@s
Geth rakudo/windows_latest_azure_pipeline: ca31fd649b | (Timo Paulssen)++ | azure-pipelines.yml
Compare-Object logs before and after
12:58
rakudo/windows_latest_azure_pipeline: 0bd9362def | (Timo Paulssen)++ | azure-pipelines.yml
is this how i have to handle spaces?
13:01
rakudo/main: fb7d3e7fd4 | (Elizabeth Mattijsen)++ | lib/RakuAST/Deparse/Highlight.rakumod
RakuAST: work around an issue in RakuAST wrt %_

The logic about using the implicit slurpy hash in a method, appears to be still not working 100%. Work around it by giving the slurpy a different name
13:04
rakudo/main: 6888e97725 | (Elizabeth Mattijsen)++ | 2 files
RakuAST: initial stab at SafeRakuDoc

This introduces a RakuDoc::To::SafeRakuDoc module, that will usually be invoked with the --rakudoc commandline argument:
   $ RAKUDO_RAKUAST=1 raku --rakudoc=SafeRakuDoc filename.raku
... (11 more lines)
13:09
lizmat patrickb: ^^
patrickb Wow, that was fast. Thanks! 13:13
lizmat it was not a lot of code sofar 13:15
github.com/rakudo/rakudo/blob/6888...akumod#L23 13:17
afk& 13:18
patrickb Bikeshedding: Do we actually want to call this "SafeRakuDoc"? Given that RakuDoc is safe as long as it has no Raku stuff mixed into it, it feels a little off. Clean or Pure seem like clearer prefixes. Or turing it around: MixedRakuDoc / DirtyRakuDoc. 13:21
Geth rakudo/windows_latest_azure_pipeline: 62bebfa680 | (Timo Paulssen)++ | azure-pipelines.yml
look at the WER localdumps registry settings ...
13:25
ab5tract I agree with the bikeshedding 13:49
I almost wonder if it would make sense to have ā€œsafeā€ be _the_ RakuDoc 13:50
And then the one with evals could be ExtendedRakuDoc or something
Just spitballing here 13:51
Geth rakudo/azure_improvements: 3e8c1ac65b | (Timo Paulssen)++ | azure-pipelines.yml
try caching location of devShell.dll

should shave off multiple minutes from every windows build
14:07
rakudo: timo++ created pull request #5642:
Try improving the azure pipeline (focus on windows builds)
14:10
coleman bike shed colors: RakuDocExec, RakuDocEval, ... 14:25
RakuEvalDoc... 14:26
Geth rakudo/azure_improvements: 59ba9a1875 | (Timo Paulssen)++ | azure-pipelines.yml
try caching nqp+moar or just moar
rakudo/azure_improvements: 03ece384b1 | (Timo Paulssen)++ | azure-pipelines.yml
try caching nqp+moar or just moar
14:28
rakudo/azure_improvements: 422db83ecc | (Timo Paulssen)++ | azure-pipelines.yml
dubquotes for all!
14:30
14:41 [Coke]_ joined, [Coke]_ is now known as a, a is now known as Guest5605
Geth rakudo/azure_improvements: 329959c5bb | (Timo Paulssen)++ | azure-pipelines.yml
fix oversight in cache handling
14:44
14:44 Guest5605 left, [Coke]_ joined 14:52 [Coke]_ left
Geth rakudo/azure_improvements: b739117ec7 | (Timo Paulssen)++ | azure-pipelines.yml
ensure nqp_git_rev is in cache key
14:55
timo dev.azure.com/Rakudo/rakudo/_build...ew=results - does anybody have access to cancel the test job here? 14:56
patrickb Canceled. 15:02
timo thank you
it's a little bit frustrating that i can't send a cancel request from github
Geth rakudo/azure_improvements: 4c2bc97b11 | (Timo Paulssen)++ | azure-pipelines.yml
copy-pasto in pwsh_dev code
15:04
timo sorry, can i ask you to also cancel the dev.azure.com/Rakudo/rakudo/_build...ew=results and dev.azure.com/Rakudo/rakudo/_build...ew=results jobs? patrickb <3 15:05
Geth rakudo/azure_improvements: 51e97fcb69 | (Timo Paulssen)++ | azure-pipelines.yml
another syntax fail
15:07
timo dev.azure.com/Rakudo/rakudo/_build...ew=results and also this
oh i already gave that
dev.azure.com/Rakudo/rakudo/_build...ew=results actually this one
15:42 [Coke]_ joined
timo dev.azure.com/Rakudo/rakudo/_build...ew=results if you have a moment to cancel yet another pipeline run ...? 15:43
Geth rakudo/azure_improvements: 8793ba1010 | (Timo Paulssen)++ | azure-pipelines.yml
where_is_devshell has to be in a folder

the Cache task can not work with single files
15:44
15:45 [Coke] is now known as Guest1428 15:46 [Coke]_ is now known as [Coke]
lizmat sorry, but to me "unsafe RakuDoc" is just Raku code 15:47
15:54 lizmat_ joined 15:56 evalable6 joined, bloatable6 joined 15:57 tonyo1 is now known as tonyo, greppable6 joined 15:58 sivoais_ joined, andinus2 joined
timo oh god i hope mv-ing the restored-from-cache folder away after restoring from cache doesn't cause a failure at the end when it for some reason tries to upload the cache? 16:00
ab5tract lizmat: well yeah.. the thing that makes it unsafe is that it executes code 16:02
I think opting in to that makes a lot more sense than opting out
timo Cache with fingerprint `"$(BACKEND)"|"downstream"|"--relocatable"|"ubuntu-20.04"|"2024.08-6-gac82e446f + 2024.08-2-gc450b5b96"|"downstream"|"--backends=moar --relocatable"` already exists. 16:03
phew.
16:03 camelia left, lizmat left, bloatable6__ left, greppable6__ left, evalable6__ left, sivoais left, jjatria left, andinus left, lizmat_ left
ab5tract timo: I was worried that you had successfully predicted your own future there 16:03
16:04 lizmat joined
timo same 16:05
16:10 jjatria joined
Geth rakudo/main: 866eec3705 | (Elizabeth Mattijsen)++ | 4 files
RakuAST: rename ::To::SafeRakuDoc to ::To::RakuDoc

And make that the default for the --rakudoc command line parameter so that you can do:
   $ RAKUDO_RAKUAST raku --rakudoc file.rakumod
to get the (safe) rakudoc of a file
16:20
16:41 andinus2 left, andinus joined 17:01 camelia joined
patrickb lizmat++ agreed 17:18
timo: I guess someone else did the honors... 17:22
timo no i think it just finished 17:23
patrickb k
Geth rakudo/azure_improvements: a07eb5a3e3 | (Timo Paulssen)++ | azure-pipelines.yml
make sure we don't throw out cached folders

doesn't seem to be a problem immediately, since when there is a cache hit, the cache job will just say
  "there is already a cache for key blahblah"
instead of trying to upload from an empty folder and erroring the whole pipeline.
But for the case that it goes wrong for some reason, let's make sure there's sensible data to re-cache there.
17:25
rakudo/azure_improvements: 65597e14c3 | (Timo Paulssen)++ | azure-pipelines.yml
Informative(?) output for devshell location caching
rakudo/azure_improvements: d7e211d736 | (Timo Paulssen)++ | azure-pipelines.yml
debug output of env vars near the start
patrickb jjatria: Did you notice gitlab.com/raku-land/raku-land/-/issues/61 ? 17:26
Geth rakudo/azure_improvements: 6fea15fa90 | (Timo Paulssen)++ | azure-pipelines.yml
forgot the -r for cp, that's rare
17:31
17:34 Guest1428 left 17:42 [Coke]_ joined
Geth rakudo/azure_improvements: 5bfe292893 | (Timo Paulssen)++ | 2 files
try out for-azure-pipeline-log stuff
17:59
rakudo/azure_improvements: dcce9b01a2 | (Timo Paulssen)++ | azure-pipelines.yml
tag rakudo builds with the used nqp and moar git revs
18:42
19:15 [Coke] left 19:36 [Coke] joined
timo should we keep attempting to run the rakudo-jvm build even though it always runs into the memory limit of the runner? 20:07
[Coke] the pipeline? I'd disable it until we can make it work again and then we can leave it on again 20:08
otherwise we're just throwing CPU away
timo the nqp pipelines also all build and test rakudo in addition to building and testing nqp 20:10
[Coke] ... in addition to the rakudo pipelines testing rakudo? 20:12
(I'm not entirely familiar with how this is setup)
timo yeah the rakudo pipelines test rakudo and there's also a job variant that also runs spectests 20:19
20:19 sena_kun joined
[Coke] so having nqp does it seems to be overkill - seems like rakudo should be testing against the latest version bump of nqp, not having nqp test the latest version of itself **and** rakudo. 20:20
... unless we're relying on that to say it's safe to *do* the bump 20:21
(but I don't think we are)
timo well, rakudo will only be tested with a given nqp version if there is a bump commit to make it so 20:27
right
Geth rakudo/azure_improvements: d28ca64124 | (Timo Paulssen)++ | azure-pipelines.yml
try cleaning the matrix up with template expressions
20:52
rakudo/azure_improvements: 04b666349e | (Timo Paulssen)++ | azure-pipelines.yml
try cleaning the matrix up with template expressions
20:54
rakudo/azure_improvements: 331206ec5f | (Timo Paulssen)++ | azure-pipelines.yml
is this how to do it?
21:13
rakudo/azure_improvements: 978bdaa0bc | (Timo Paulssen)++ | azure-pipelines.yml
is this how to do it?
21:15
timo this shit's not working as advertised i feel like ... 21:16
21:37 sena_kun left
timo oof. 21:42
i can just throw all of that last stuff in the trash
21:49 El_Che left 22:00 sena_kun joined 22:31 sena_kun left
Geth rakudo/azure_improvements: c967f29442 | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:18
rakudo/azure_improvements: e48a4e119c | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:20
timo i want a "don't annoy the people on irc with this" thing for this branch 23:21
Geth rakudo/azure_improvements: c7005da118 | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:22
rakudo/azure_improvements: 14aac1d5fb | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:27
rakudo/azure_improvements: 0b7baa4ad8 | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:41
rakudo/azure_improvements: 9793b41ffa | (Timo Paulssen)++ | azure-pipelines.yml
try to template expressions all the things
23:44
rakudo/azure_improvements: d8c403e890 | (Timo Paulssen)++ | azure-pipelines.yml
always stop JVM before the "build rakudo" step

it's very slow and always runs out of RAM
23:52
rakudo/azure_improvements: 7620b7c90b | (Timo Paulssen)++ | azure-pipelines.yml
always stop JVM before the "build rakudo" step

it's very slow and always runs out of RAM
23:53