Fire is step THREE! | github.com/perl6/toolchain-bikeshed | Channel logs: irclog.perlgeek.de/perl6-toolchain/today | useful prior art: metacpan.org/pod/CPAN::Meta::Spec
Set by moderator on 15 January 2016.
02:30 MadcapJake joined 02:48 ilbot3 joined
moderator Fire is step THREE! | github.com/perl6/toolchain-bikeshed | Channel logs: irclog.perlgeek.de/perl6-toolchain/today | useful prior art: metacpan.org/pod/CPAN::Meta::Spec
03:54 moritz joined 05:43 MadcapJake joined 07:09 domidumont joined 07:14 domidumont joined 07:25 domidumont1 joined 07:32 domidumont joined 07:37 domidumont1 joined 07:53 FROGGS joined 09:39 leont joined 10:49 FROGGS joined 12:37 llfourn joined 13:38 llfourn joined 13:51 nine_ joined, moritz_ joined, jdv79_ joined, masak_ joined 13:59 stmuk joined 14:02 FROGGS joined 14:03 PerlJam joined 14:05 autarch joined 15:43 FROGGS joined
ugexe nine: github.com/ugexe/rakudo/commit/31a...f0070785c2 this spot causes random precomp errors on windows when doing a force install over a previous install (see: ci.appveyor.com/project/ugexe/zef/...13#L1120). Doubt this is an ideal solution though 15:46
it is random in that it doesnt always happen, not in which file/dir causes the problem 15:47
im not sure if `try .unlink` avoiding the failure could have ramifications during a subsequent run if the files that failed to delete somehow get used 15:58
xdg ugexe, why doesn't it check the return values of unlink? What if it fails? 16:08
(or does it throw if it fails?)
ugexe because what is it going to do? 16:09
windows likely isnt going to let go of the file until the process is done
but thats the part of the it above about 'doubt its an ideal solution' though
xdg IIRC, you can't unlink it if there are open handles 16:11
ugexe right 16:12
but what would you do with a caught exception during a *force* install where an unlink fails? you could keep the program indefinitely until it succeeds, or you can fail in a way that doesn't match the option of "force" 16:16
keep the programing trying to unlink^ 16:17
and just having a command prompt open to a directory that a perl6 script is trying to delete can often cause the same problem, not just an open handle from within rakudo itself 16:18
xdg if you can't unlink it, the win32 api can schedule it for deletion on reboot. 16:20
but I was actually thinking that if unlink fails, you could get an error about which file is failing and figure out why 16:21
ugexe if you are force installing a module its going to go into the same sha1 path name that is trying to be deleted
xdg In Perl 5, a __DATA__ section that wasn't closed was usually the culprit.
maybe sha1's are the right thing, then. 16:22
s/are/aren't/
sha1.n? And always take the highest n?
ugexe dunno if that would work. if sha1.1 and sha1.2 get used by different modules but use the same identity it seems like there would be some ambiguity somewhere 16:27
tony-o the api idea is a good one, but it seems like it should happen lower in the food chain somewhere 16:28
ugexe it happens both low and high. the compiler would search for identities including the :api just like it does with :ver. but then your module could look like `package GitHub { class API:api<1> { #`( v1 api access); }; class API:api<2> { #` (v2 api access ); };` 16:32
and then `use GitHub::API:api<1>` or `use GitHub::API:api<2>` 16:33
tony-o i mean the win32 api for unlinking on reboot, that should happen lower in the foodchain rather than being a part of Precomp
or the option to do that should be present lower in the foodchain, if it isn't there already
ugexe well sometimes you cant wait until reboot
tony-o for unlink?
ugexe yep
tony-o for the situation in particular or in general? 16:34
are you talking about reinstalls?
ugexe in the precomp problem above its using CU::R::I.install(:force). the module it installs will have the same sha1, so its going to go to the same path that already exists. but if that path can't be deleted first then how do you install the new dist to the same folder? 16:35
in this case unlink at reboot would not help
tony-o that sounds like a die situation or some kind of hacky work around
like taking the highest N in xdg's solution above 16:36
ugexe but why would you expect 'die' when you pass something like :force
highest N means you cant just generate the path, and involves constantly grepping the directory for highest N
tony-o and then checking for .Ns and fixing it when it's possible rather than *now*
which has its own set of problems 16:37
ugexe how do you test something that isnt fixed now?
tony-o if there are .Ns then use the highest .N, on start up if there are .Ns then unlink all but the highest and then mv the remaining .N to whatever the sha1 should be and overwrite the existing 16:38
it's hacky, i'm not advocating for it
it's doable, though
ugexe that works but introduces lots of slow downs related to constantly looking up available files instead of know the path beforehand and just checking for existence 16:39
tony-o agreed
what is the other option, to wait? 16:40
ugexe or just `try` to delte as much as you can
tony-o shell an 'rm -Rf ' ?
ugexe this is windows, and any shell command will face the same problem 16:41
xdg stepping back -- if the file is identical, why replace it?
ugexe xdg its the same identity, not neccesary the same contents
xdg what does that mean?
tony-o you can force delete files with an open handle in windows via shell
ugexe identity is name, version, auth, api 16:42
tony-o xdg: the sha1 of the name/ver/auth/api/etc is the same, not the contents of the file
ugexe sha1 is of these 4 fields, not of the contents of files
tony-o so the path is the same, but the file isn't - the md5sums would differ
xdg (dare I ask why bother to sha1 those?)
tony-o so that you don't have to deal with special chars in the path
ugexe to make the names same to use as a file path 16:43
tony-o and that ^
xdg hmm. seems like overkill to me.
whatever
tony-o shell("handle $filename"); shell("handle -c $handle-id -p $process-id");
xdg: seems that way but it's one of the better/safer options 16:44
xdg indirect? Have a table of name/ver/auth/api SHA1 mapping to table of content SHA1s and put file under content SHA1?
ugexe that was done before. its slow
xdg slower than sha1.n? 16:46
tony-o ugexe: ^^ you can shell close files that way even with open handles in windoze
xdg: yes.
not on a small magnitude but once you get a decent number of modules involved, it's very slow
ugexe sure, but for all i know that precomp file is actually still in use 16:47
trying to install over itself
xdg I'd be shocked if a single map lookup is "slow" compared to the load time of the module itself.
tony-o that seems like a disparate comparison 16:48
xdg or do you mean loading the table is slow?
ugexe you'd be surprised if you cloned a 6 month old rakudo and installed ~5 modules
~50
tony-o loading the table is slow
xdg don't load it. binary search it and cache the results. 16:49
(pay the sort cost when modules are installed)
tony-o cache it in what, exactly?
xdg the runtime
tony-o you still have to load that every start up 16:50
which is the slow part
xdg but maybe you dno't need it again. nevermind.
ugexe anytime you start up and use a module rather
xdg on posix, you could do it with symlinks
Windows... ugh 16:51
they have a thing like that
junctions or something
Can rakudo unload the precomps? 16:53
tony-o even if it can you can't guarantee that there isn't another process using the precomp file 16:54
the symlink idea is interesting
xdg tony-o, you could even fake it on windows. Save content SHA1 inside a file named after name/ver SHA1. 17:01
you're then slurping 50 files for 50 modules instead of loading one database... so not sure that saves you much 17:02
ugexe thats what Zef::ContentStorage::LocalCache does fwiw 17:03
xdg How was the table done before? SQlite?
ugexe it was a json file... with way too many levels of structure
xdg gaaah!
all you need is a text file. Two columns: name SHA1 and content SHA1. Sorted on name SHA1. Do binary search. 17:04
tony-o sqlite is too much dependency, i have some binary tree code at home somewhere that i was working on to make zef faster
don't even need the content sha1
xdg all you need is the equivalent of Perl 5 Search::Dict
tony-o you're just moving the problem, though
xdg I was thinking content sha1 to ensure different contents can be written without conflict. 17:05
tony-o multiple installations at once will cause issues
xdg only if you don't account for concurrent access
ugexe an alternative, faster lookup table format has been discussed recently but nothing beyond that
tony-o concurrent writes?
xdg shared lock to read and search. Exclusive lock to write a new sorted list.
ugexe all the dist/precomp stuff uses locks before writing 17:06
tony-o that still just moves the problem to a locked lookup table file rather than the precomp file
xdg locked lookup table isn't ever deleted, though.
you might still have concurrency issues on windows -- not sure if you can have multiple writers open. 17:07
tony-o it's the same concept, what do you do when that file is locked for writing? die? wait?
what if it's locked by a hung process? what do you with that?
xdg welcome to concurrent systems programming 17:08
my point is that by moving the problem to user-space, you have more control than what the OS might give you 17:09
tony-o those are rhetorical questions
xdg Instead of locking the table, use a lock file and lock that. Then if you've waited "too long", delete the lock file and start over.
there's no free lunch 17:10
ugexe thats what is already done (except for the wait too long part)
xdg anyway, I have to get back to work. I hope there are some useful ideas you can build on. 17:11
18:00 FROGGS joined 18:03 domidumont joined 18:12 leont joined 18:16 flussenc1 joined, b2gills1 joined 18:17 tony-o_ joined
tony-o_ xdg: i'm familiar with the concepts, i'm simply pointing out that moving the problem isn't actually resolving it 18:47
and i realize there isn't a fix all for the issue 18:48
mst tony-o_: you're condescending to one of the longer standing members of the perl5 toolchain team 18:58
tony-o_: maybe try assuming he's also familiar with the concepts and not saying dismissive things like "that's just moving the problem" as if it isn't obvious what's what we're doing, and that sometimes moving it to a better spot is the right answer 18:59
tony-o_ mst: i thought i acknowledged moving it might be better, and i'm not trying to be condescending towards anyone. so, my apologies if it's coming across that way. 19:12
mst tony-o_: you have an unfortunate tendency to come across as rather more hostile than you intend, which is responsible for a fair number of your suboptimal conversational outcomes. note that I mean that to inform, not to attack - I have exactly the same problem myself :) 19:13
tony-o_ no worries, i don't mind feedback on my methods of communication. thank you 19:15
mst xdg: might I suggest that applying the filters you've developed for tolerating me may help make talking to tony-o_ more productive :) 19:16
19:21 leont joined
xdg mst, thanks for the tip. So applied. 19:28
tony-o_, re "moving the problems", it's good to be clear what they are: (a) replacing an open file; (b) concurrent module installation. 19:29
b2gills mst: I think you are calling the kettle black
xdg Solving (a) by transforming it to (b) might be an improvement, since there has to be a solution to (b) anyway. 19:30
b2gills, at least mst knows his color, even if it doesn't always moderate his comments. :-)
b2gills
.oO( I should have finished reading before inserting foot )
masak in my recent but limited experience, constructions like `try .unlink` are traps waiting to happen 19:32
b2gills What I often do is write out my response then don't post it for a minute or more. Either I then edit to be nicer, or I just delete it.
mst b2gills: I am, quite consciously so, but it looks like you got to the part where I said that already :) 19:33
b2gills mst: I still think you're awesome 19:34
tony-o_ xdg: i agree with you, fwiw. i think that file locking and the table might be a much cleaner problem if we can resolve the speed. like you, i believe it's doable 19:37
s/problem/solution/
i'm much more aware of how my words come off in person because i can see peoples' reaction to my speech and i can modify it a bit. typing is a little more difficult because i can only get my reaction to what i type unless, like mst did, someone tells me that what i'm typing sounds condescending or harsh 19:39
mst I'm fortunate in that there are quite a few people who know me well enough that if I start being a prat, a /msg window will light up containing "dude, you're doing it again" or so 19:41
xdg When I wrote CPAN::Common::Index::Mirror, I used Search::Dict on the 02 packages files (170k lines or so). It was pretty fast -- far, far better than loading into memory.
will it be faster than directly accessing the FS, no 19:42
tony-o_ as long as it isn't what it was then i think we're okay. people were reporting closer to 90s increased start up time on a 'use Module;' 19:43
i can work it on the plane ride home, i need to look at the code more closely, though.
xdg ooh, brainstorming again!... Have the name SHA1 be a *directory* containing precomps
usually just one. if more than one, take the "latest" one 19:44
(except you're screwed on FAT32 because of 2-second file timestamps)
mst oh, there's an interesting question. will win32 let us *rename* a directory? 19:45
xdg I doubt it, but I don't really know.
b2gills # Back in the day if you renamed a dir on Win9x it would keep it's 8.3 short name if it shared the same first 6 chars 19:47
xdg mst, I just checked and no. "Permission denied" 19:52
won't let you rename a dir if there's a handle open to a file in it
what i don't like about putting precomp files in a directory is that there's a race to choose the right one 19:53
stepping back -- is there a "don't use precomp" mode for rakudo? If this is (or could be), then just run that way for installation 19:54
`#!rakudo --no-precomp` at the start of panda or whatever
mst what bothers me is the case where there's a running daemon or something with it open 19:55
xdg right
"Error: can't upgrade Foo::Bar while it's in use. Please close all running programs..." 19:56
[Coke] you can add "no precompilation" in the source.
but that's probably not quite what you need. 19:57
ugexe thats what my problem is. but then a module installer cant upgrade itself
xdg I meant conditionally for a given invocation, but mst is right it doesn't help if some other process has the procomp loaded
mst I think having a directory, and then an index into that directory, thereby reducing the problem to making sure the index doesn't get left open, is going to be the only way to win
and then something can periodically nuke not-in-the-index-anymore precomps
xdg Stepping back again: what's happening with the precomp that it has to stay open?
Here I freely confess my ignorance of the details of rakudo. Sorry. 19:58
b2gills really that's details of MoarVM
mst actually, yeah, if we could make that go away that'd be nice
except
xdg right. that.
mst I'm not sure I -want- to be deleting a precomp that a running rakudo is still using 19:59
ugexe that was my concern as well
mst because, well, because what if your daemon exhibits a bug ... and the precomp you'd need to repro the bug went away
xdg if it's like a shared library, no. If it's a cache of compilation, then no big deal?
mst I dunno if that's a realistic problem
b2gills I suppose it would depend on if the .moar files are a in-memory data structure or just a serialization 20:00
leont thinks that's not something to worry too much about
xdg downgrade to prior version to repro
mst well, it also includes what versions of all its dependencies were used
xdg o/ leont
leont xdg: good to see you here too :-)
xdg I'm avoiding thinking about version numbers.
mst a fine and long standing traditional activity for chainers ;) 20:01
leont b2gills: even if it's in-memory data structure that's mmap()ed, that doesn't have to matter if you're careful
tony-o_ that sounds like a good solution xdg
oops, typed that 20 minutes ago
xdg :-) 20:02
mst I really want to look at seeing if we can cache dependency lookup information in precomp in a way that allows us to be more selective about blowing caches
20:25 Kassandry joined
flussence if I could change one thing about the precomp system, it'd be to use flat files + directory trees instead of parsing JSON ($PREFIX/dist/*). Or at least, less of the latter. 20:35
(CURLI is better than that horrible json-inside-tsv thing panda used, at any rate) 20:36
leont xdg: OT but you might appreciate my Path::Iterator module :-)
json-inside-tsv? That's perverse!
flussence the fun part was when it blew up because the module that wrote the json decided one day it'd start formatting with newlines by default :) 20:37
tony-o_ lol
leont Ouch 20:38
flussence wondering if there's any useful ideas we could borrow from distros' package DB designs 20:45
mst nix might be interesting prior art in that respect 20:46
ugexe just ask yourself What Would Microsoft Do? clearly the answer is to just use xml 20:51
21:05 hankache joined 23:29 llfourn joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!