|
00:00
rba left,
rba joined
07:38
rnddim joined
07:39
rnddim left,
rnddim joined
07:40
ShimmerFairy left
07:59
rnddim left
08:00
rnddim joined
14:21
Techcable left
14:22
Techcable joined
17:17
tbrowder__ joined,
SmokeMachine_ joined
17:24
tbrowder left,
SmokeMachine left,
SmokeMachine_ is now known as SmokeMachine,
tbrowder__ is now known as tbrowder
21:13
timo left
21:46
timo joined
22:16
MasterDuke joined
|
|||
| MasterDuke | timo: i have a very rough branch that uses simdutf in MVM_string_utf8_decode. however, it isn't really any faster (or smaller) because there are a lot of cases where it can't be used and it just falls back to the existing code | 22:18 | |
| tellable6 | 2026-07-18T21:39:25Z #raku-dev <ugexe> MasterDuke: Thanks, I have a branch adding some dead code elimination now but it’s behind like 4 other branches I have yet to review and push | ||
| timo | what cases are those? | ||
| MasterDuke | any where `MVM_string_convert_utf8_to_latin1` errors out | ||
| so i think any codepoints/graphemes/whatever > 127 | 22:19 | ||
| timo | that sounds like it would be anything where utf8 has more than one byte for a codepoint? | 22:20 | |
| MasterDuke | github.com/MasterDuke17/MoarVM/tre...th_simdutf has the code, but it's not really usable as is | ||
| i'm not sure if this is worth trying to finish up/make viable as a real PR | 22:21 | ||
| i could try to replicate what i have for `MVM_string_utf8_decodestream`, which would get it used in a couple more situations | 22:22 | ||
| but if it isn't faster, there's no real point | |||
| so i'm wondering if you have any ideas that could make this worthwhile | 22:23 | ||
| i.e., i'm not sure how our \r\n handling may interact | 22:30 | ||
| s/i\.e\./e.g./ | |||
| or if there are any different libraries that'd be useful? or is our NFG unique enough that if we want to get faster we have to do it (mostly) ourselves (write our own simd directly)? | 22:33 | ||
| timo | yeah, I'm really not sure if any library would happen to do the strange things we do | ||
| presumably none of the "utf8 parsing" libraries do any normalization | 22:34 | ||
| and we do more normalization than anyone else | |||
| MasterDuke | i haven't looked at what swift uses, since i think they're the next closest | ||
| i just saw github.com/dzfrias/xxUTF, but haven't actually looked at what it offers yet | 22:35 | ||
| timo | they put the "move from one grapheme to the next" logic where-ever an utf8 string is moved through | 22:39 | |
| MasterDuke | it seems like swift implements whatever they do themselves in swift | 22:42 | |
| timo | xxutf might be interesting. difficult to say | 22:49 | |
| MasterDuke | would it make any sense to use xxUTF to do NF[^G] normalization and then (hopefully very quickly) post-process its results? | 22:50 | |
| is it possible to quickly find where in a string/buffer NFG would differ from NF[^G]? maybe make more strands than we do now by using those libraries for parts of the input, and then doing our NFG stuff for the remaining parts? | 22:53 | ||
| or how difficult would it be to add NFG to xxUTF? | 22:54 | ||
| dinner & | 22:59 | ||
| timo | even after you use xxUTF to normalize a string into NFC, that's going to be an utf8 string afterwards too right? | 23:00 | |
| utf8 has no way to represent NFG codepoints | 23:01 | ||
|
23:03
MasterDuke left
|
|||