Fire is step THREE! | github.com/perl6/toolchain-bikeshed | Channel logs: irclog.perlgeek.de/perl6-toolchain/today | useful prior art: metacpan.org/pod/CPAN::Meta::Spec
Set by moderator on 31 August 2016.
01:48 ilbot3 joined
moderator Fire is step THREE! | github.com/perl6/toolchain-bikeshed | Channel logs: irclog.perlgeek.de/perl6-toolchain/today | useful prior art: metacpan.org/pod/CPAN::Meta::Spec
04:48 JimmyZ joined 05:04 pnu_ joined 05:05 tbrowder joined 06:39 domidumont joined 06:45 domidumont joined 06:50 nine joined 06:57 domidumont joined
nine Well however this will end up, at least a module author won't have to do anything special for getting the dependencies installed. That's one step forward from Perl 5 already. 07:12
lizmat nine: this is quite the time to be giving a presentation about Perl 6 module development 07:32
I hope I won't be too outdated 07:33
nine Well this whole area is behind core rakudo by at least a year, maybe two. Things are still very much in flux. 07:36
lizmat: but if you like, we can go over your material this evening. Maybe we can at least include pointers to likely changes.
lizmat nine: sounds like a plan! 07:37
08:19 leont joined 09:02 leont_ joined 09:03 nine joined, jdv79 joined 09:27 domidumont joined
nine If we are concerned about running build steps in the installer's process, we can just run them in a different process still without needing a user supplied script: gist.github.com/niner/b23726cdef53...fc616702c4 10:37
With this interface it's still gonna be easy to move to out-of-tree builds for example without dists having to catch up. 10:39
mst nine: not sure what you mean by "won't have to do anything special" 11:09
nine What I was trying to say is that we'd still have a lot more control over the build process on the installer's side. That means fewer surprises and fewer terrible hacks we'll have to stay compatible with. 11:10
mst to be entirely honest, other than "oh gods why does Module::Build accept 27 different permutations of its options" and "oh gods PREFIX" 11:11
nine Though I very much like how "class MyBuilder is Dsitribution::Builder does GitCloneRole" looks like, I just can't convince myself that this is the safest path.
mst driving perl5 builds from the installer side really isn't that awful these days
nine: huh?
nine: that was the plan you argued me into wasn't it
*confused* 11:12
nine Well, I'd suggest being able to specify roles for the builder in the META data, while your suggestion is to have a small configure.pl6 that contains this line.
What I fear is that there's nothing at all making sure that the configure.pl6 is indeed small and simple and doesn't contain horrible code that would be much better to have in the installer proper. 11:13
mst that's why I said you should use inc/MyBuilder.pm for it
so it's clearly *not* a configure script 11:14
also, this is supposed to be a perl
we're not supposed to be afraid of giving users rope
we're just supposed to try and stop them hanging anybody *else* with it by accident
I can pretty much guarantee you that at some point somebody's going to need a code snippet during their build process that's module specific, and forcing them to do two complete releases just so the first one supply three lines of build code is not really very fair 11:16
also, remember that it could contain code that a *descendent* of belongs in the installer 11:17
nine Oh I fully intend this to have plenty of points for simple extension and hooks. Just like systemd supports ExecStartPre and friends. I'd just like to see the build process itself driven by declarative META data instead of arbitrary Perl 6 code 11:18
mst inc/ is a place for small experiments that you *don't* want people to install independently *because* it's a hack
so would I, *eventually*
for *most* cases
but the escape hatch is important
nine Like "builder": "inc/MyBuilder.pm6" or something 11:19
DrForr I had to crash out for a bit last night - Did the install script thing get straightened out? IIRC Eduardo was just confused over how env vars worked.
mst precisely
nine burried a bit deeper in the docs so only desparate users will discover it
mst only people doing something weird should 11:20
like MY::postamble is only for 'something weird'
but, like, consider that, p3rl.org/Alien::Base has all sorts of neat options and stuff 11:21
but 'a giant postamble' was the way to get a viable Alien::Rakudo dist
I would *prefer* that most perl6 dists look more like an Alien::Base configuration than my epic hack
but I would also think it would be sad if we managed to make it so perl6 dists can't do something as insane as Alien::Rakudo at all
nine: note, I'm ok with 'builder_roles:' as well as 'builder_class:' 11:23
nine Oh definitely
mst I just also think that at some point people will need three lines of custom code
or to parameterize a role 11:24
or
and I want the escape hatch
and you can also document 'inc/MyBuilderRole.pm6' that way
to encourage people to have a small custom role
not a full custom class
11:24 nadim joined
mst basically, make the right thing sufficiently easier to do that people default to doing that 11:24
but make sure that once one falls outside the normal moral constraints, things are still customisable 11:25
because there *will* be things that require per-dist hacks that aren't worth releasing separately
(sorry for wall of texting, but I think if we get the aesthetic right everything else will fall out from there, and I have Opinions about the correct aesthetic here ;) 11:26
nine Also as it turned out, we were pretty much on the same page anyway. Just needed a bit of more text on the screen to figure that out :)
nadim I didn't think this day would come but I agree with mst 11:30
declarative systems work fine as long as what you declare is enough to build everything.
that works if there are people working non stop at keeping the build system up to date with the development 11:31
mst also, we won't initially know *how* to declare things
nadim after that you start making the declarative system "smart" and flexible, and your done with it.
mst also, if you do that, it stops being declarative and starts being a small underdesigned shitty scripting system
nadim IMHO, based on a long experience of big systems, you need a flexible system to start with, on top you build a simple declarative system 11:32
people with simple needs keep to that and other us whatever is under. with some luck, a crazy or two work a bit on the declarative system now and then. 11:33
mst right, and by having configure_requires we can have multiple declarative systems with plugins 11:34
and then the inc/ escape hatch for dist specific stuff
nadim but first things first ... REQUIREMENTS
as long as those are not down, this is just an exercise in hacking for hacking sake 11:35
nine likes hacking ;)
nadim there must be other toys than build systems to hack on 11:37
mst nadim: nine's list of existing functionality is a start towards that
nadim I have worked on a build system with 2000 requirements, not a single one was at that high level. 11:39
those are requirements for what kind of result one wants, and they are good to have, but how is the system to behave, who will use it, what error are expected, how are they reported, ...
IMVHO, the first error is to rely on make 11:40
mst on a long enough timeline, we definitely shouldn't *need* make 11:41
nadim seriously, on any time line there should be no make involved at all
if some module wants to use make because their build is already in make (or other), then so be it 11:42
mst that's what I meant
can we, for the sake of argument, both try assuming that we've both done lots of build system hacking and don't need 'seriously'-ing? :)
nadim no, seriously means that I think deep and long and seriously think what I write, what you have done or not is your business 11:43
nine Now that I think about it, even Inline::Perl5's perlopts variable has a place in common build system code. Because there may well be modules that carry a Perl 5 XS part that needs compiling. Maybe even Inline::Perl5 can supply a Builder role for that key. 11:44
nadim I don't know what you or other have done, and I guess you will "seriously" tell us
mst ...
nine The important part being not perlopts but composability of the build system
mst ok, english idiom-wise, "seriously, ..." in that form comes off as "i am giving you a 101 statement because clearly you don't understand the basics", I'm just asking you to try not to be condescending
nine didn't know that 11:45
mst it would be kind of bizarre for us to go from agreeing for once to disagreeing because *you're* being impolite - that's my failure mode, damnit, don't steal it :P
nadim mst: enough 11:46
let's continue on this and not put so much on the form, please
nine Frankly, I've just gotten tired of grep Makefile.PL | cut -d "'" -f2 just for getting at the list of dependencies of our code. That's where my drive for parseable build data comes from. 11:48
mst heh, yes 11:49
similar to PREREQ_PM parsing out of the Makefile for EUMM
mst shivers
we're probably going to need a MYMETA6.json eventually
nine has never understood what all those files are supposed to be or do
mst since OS-specific dependencies have enough dark corners that while we can declarative-ify most of them, there'll still be edge cases
MYMETA is "meta, but after the configure phase" 11:50
to handle dynamically selected dependencies
nine so it is generated?
mst yes
nine So it's actually part of the interface between the installer and the custom build code? 11:51
mst basically MYMETA is "META, but with system-specific stuff figured out by the configure phase merged in"
nadim what is that we are building that has os specific dependencies so complicated that it would wait for a P6 build system and not already have its own configuration system?
mst nine: so, basically, the perl5 installation protocol is 11:53
'install configure_requires from META.json'
'run Makefile.PL'
'get build_requires, test_requires, and runtime_requires from MYMETA.json and install those'
'do rest of build'
having the MYMETA deps be different to the META deps isn't something you need most of the time
but when you do, it's pretty much essential 11:54
nine As a packager I really, really, really want to be able to extract package dependencies for openSUSE without having to execute Perl (6) code. And as a user I'd really like a simple way to just install them. And I'd very much prefer to get rpms of all my Perl 6 dependencies installed if available.
mst yes. hence why I said 'eventually' 11:56
nine mst: an equivalent would be for the Builder to just modify $dist.meta in the $builder.configure() call, before the installer starts installing dependencies.
mst it's another escape hatch
but to give an example, let's say you have a system that wants to depend on a module that's designed to fettle the init system
nine At the very least, through this dicussion, I'm finally gonna learn how the Perl 5 build systems work :) 11:57
mst you can absolutely do declarative to find out the default per OS
nadim nine: I have more questions. Why would you not like to have to run P6? because you don't want to install it (you as anyone that is making packages)?
mst but e.g. latest debian stable comes with systemd by default, but replacing it with sysvinit is still supported
nadim nine: having rpms means tests are not run, I can see the attraction in that but that is exactly what we do not do in perl.
mst so I'd have the configure stage detect which init system is in use, and then add Thingy::SysVInit as a dep if it detects that being in use 11:58
etc. etc.
I'm not sure that's a *great* example, but I think it elucidates that there will, sometimes, be times when dynamic deps are the best way
even accepting the disadvantages
nadim or the responsibility of getting all right is on the packager and that, IMO, doesn't work so well in the long run
nine nadim: because when I run that P6 code, it does what the developer inteded. E.g. in a Perl 5 Makefile.PL we have calls like "requires 'Data::Dumper';". That's intended to tell Module::Install the requirements, so it can install it. That's useless for me as a packager. 11:59
nadim: I can get around that for a specific package easily. But if I want to automate creating rpms for all published Perl 6 modules, that's hell.
mst nine: er. that goes into META.json 12:00
nine: and then if there's system specific stuff it goes into MYMETA.json
I feel like your lack of experience with the perl5 build stuff is leading you to regard packaging it as harder than you think
I mean, there are some annoying problems in there, I'm not saying there aren't 12:01
nadim nine: if the build system has functionality to give you a rpm why bother extracting variables and creating the rpm yourself
mst but maybe not all the things you think are problems
nadim nine: this is a bit the package manager vs perlbrea/cpan problem. Which one model do you think is right?
nine mst: so for Module::Install based dists I have to run perl Makefile.PL and then read MYMETA.json. But what about other build systems?
mst nine: for any modern perl5 dist, you do that 12:02
you run either Makefile.PL or Build.PL, then you read MYMETA.json
nine "any perl5 dist" would be so much nicer
mst there are a few lying around that are pre-META-system
those you have to fall back to extracting the deps out of the Makefile or Build file, but they're a dying breed now, thankfully 12:03
yes, it would be nice if we'd pre-invented all the generation N tools before generation 1 started, but time doesn't work that way unless you're Doctor Who :D
nine It's still not exactly straight forward to get at the Perl dependencies. Much less so for external dependencies. Again we have some modules which try to deal with that. But nothing an ordinary distro person would be able to follow. 12:04
mst nine: $meta->{runtime}{requires} is a hashref of runtime deps 12:05
12:05 domidumont joined
mst not sure how much more straightforward it can be 12:05
nadim nine: please ping me when the dust has settled a bit more. I'd rather let you and mst brainstom till you melt first ;). IMO, write down some requirements, with detailled examples if you can, those can be reviewed by the rest of the community too, as well they work a bit to help.
mst and external dependencies are a wicked problem
nine Yes, they are. But I'd really like us to at least try to solve it for once. 12:06
Even if it will just get us 80 %
nadim: fair enough :)
nadim: mst is an excellent sparring partner after all :) 12:07
nadim you seem to be all over the place both of you as you've been on this longer.
yes, the best
mst nine: I would like us to maybe try it in x_ keys to begin with, since I suspect we'll need to solve it more than once before we actually get a solution 12:09
nine Mind that external dependencies may as well be Perl 5 modules, so they are probably more common than in Perl 5.
I'd be totally fine with that 12:10
mst yeswell, on a long enough timeline I intend to be writing a cpan client that handles perl5/perl6 hybrid projects as a first class concept
because I'm going to need one of those, and I like writing things like that
(this is, after all, one of the reasons for Alien::Rakudo ;) 12:11
12:12 domidumont joined
nine A reason for why I'd like at least some handling of external deps to be in right from the start is that there are actually modules for this on CPAN. I just strongly suspect that they have so little uptake, because they're not easy to discover. 12:18
mst I'm not using any of them because I've not encountered one I actually thought was any good 12:19
I do want to have a look at szreic's CPAN::Plugin::Sysdeps though
that one, finally, is one written by somebody I think stands a chance of making something that actually works 12:20
not sure which others you're thinking of
but if you tell me, I can either tell you why it won't work, or facepalm and go try it next time I have that problem, depending :D
nine Don't know any right now. Just remember encountering bits and pieces over the years 12:21
mst so do I, and then I looked inside, and then I facepalmed 12:23
I've not yet countered one where 'not easy to discover' was the problem :P
nine for you
mst right, but basically, in the case of the ones I've seen so far 12:24
people not finding them and trying to use them seemed like the better thing to me :D
nine I'm not sure. There's few things as sucky as parsing compiler errors to find out which god damn lib and -devel package I need to install. 12:26
mst why would you do that? the Makefile.PL will report "Unable to find -lfoo" or so 12:28
nine: you're really doing things the hard way here :P 12:31
nine mst: they do that if the maintainer actually cared enough to add a check 12:51
mst nine: that feature's been built into ExtUtils::MakeMaker for as long as I can remember. 12:52
nine Which many don't. Probably because they have no idea how.
mst: then why do so few seem to use it?
mst how the fuck should I know? it's in there, for at least a decade, it's documented, I found it fine
I fully expect that half our users will ignore whatever feature we add to support this too 12:53
welcome to toolchainery, you can't fix stupid
do remember I first got onto CPAN because Guile.pm's Makefile.PL didn't run unless Guile.pm was *already built and installed* 12:54
it's amazing how creatively terrible people can make their dists
nine I can't find anything like that in Module::Install's docs. 12:55
And if you tell me now that I shouldn't use Module::Install in the first place, then I'll answer that I instantly fell in love with the perlvogue idea, because like back than I still havn't got a clue which of the million options is actually considered the right one nowadays 12:56
mst I'm not going to tell you that, though had you read the mstpan articles I wrote you'd already know which options are the right ones
nine I probably have read them. I still have to remember where I've read about this topic when I start a new dist. It's again not exactly easy to discover :) 12:57
mst go to my blog. click. :) 12:58
nine Know that it's on _your_ blog in the first place
mst yes, I know that
but there isn't really an official place to put it
nine sadly
moritz suggested fix: create a perlmodauthor.pod, submit it as a patch to p5p 12:59
mst nine: there we go 13:00
nine: if you've put the libraries you're trying to link into LIBS, as documented, it'll warn if it can't find them
nine: so I've no idea wtf the Makefile.PLs that didn't do that were doing
moritz document the One True Way to author a perl module/distribution
mst nine: and Module::Install is just a wrapper around EUMM so that works via M::I fine as well
you just put it in your makemaker_args() 13:01
nine So how can we avoid creating such confusion in the first place in Perl 6 land? 13:02
mst ... dude :( 13:04
how many times am I going to have to say "it's been there for fifteen years, it's documented, it's the first thing you find if you actually read the documentation, I've no idea why people didn't do it that way" ?
presumably "put it in bigger letters in the documentation and make sure there's a tutorial" 13:05
nine Wait...does EUMM actually check if the acoompanying header files for -lfoo are installed? 13:07
mst oh, hrm, it might only check for the library itself 13:08
usually on a build machine I either have neither or both
nine Also EUMM seems to write that information only to the generated Makefile but not something readable like MYMETA.json. So again no simple way to extract it for the fellow packager. 13:11
mst I never said it was perfect 13:12
I did say that CPAN::Plugin::Sysdeps is experimenting with stuff
also there's metacpan.org/pod/Devel::CheckLib
my point is not that things aren't suboptimal 13:13
my point is only that you're dismissing it as all terrible when there's actually quite a bit of stuff there that does work, if authors use it 13:14
nine CPAN::Plugin::Sysdeps does so by having a centralized list of dependencies for CPAN packages. Can't imaging that scaling well :) Though at least it has this mapping in an extra module so it may be used for other purposes like determining dependencies for packaging 13:15
mst yes, it does so that way because it's trying to experiment WITHOUT requiring modifying all of the existing modules 13:16
nine My whole point is: creating distro packages for perl modules is a never finished task that involves a lot of manual work when it shouldn't have to. I'd like to avoid that at least in Perl 6.
mst remember that we go slowly and conservatively to avoid trouble
and my point is: you seem to be assuming perl5 just didn't bother to try, and the truth is we have but it's a fuckload harder than it first appears 13:17
nine I get that. But we're talking about someting for Perl 6 where we still can go faster and break things once or twice. We ought to make good use of this luxury. 13:18
mst sure. but charging ahead pretending other people didn't try is not a good use of it.
nine And that's why I'm discussing it here and intend to take it to perl/#toolchain as well :) 13:27
mst yeah 13:33
just you seem to be falling into the trap of "nobody else has done this yet, so that must mean they didn't try" 13:34
whereas the reality is much more "lots of people tried ... and failed"
nine So taking a step back, I guess it's a good idea to document the requirements I have for a build system * as a user * as a packager * as a module author. Then discuss it with experienced people to find out what has been tried, what has failed and what is at least promising. 13:36
mst yeah 13:38
there's a *huge* amount of prior art of the form of "yeah, we tried that, and this is why it didn't work" that we should be able to collect
also, calling it 'a build system' is an error 13:39
there's kinda three separate things here
* dependency resolution 13:40
* converting a tarball into the stuff to be installed
* installing it
the CUR API is basically handling the third one for us
well, * dependency declaration
the module needs to say "I'm going to need this stuff to build" and "once installed, I'll need these things to work" 13:41
and then the installation manager needs to deal with that
nine: I wonder if what we want is a sort of pluggable spec for dependency specs 13:47
I dunno
but it gets interesting 13:48
because of cases where e.g. you need 'libfoo compiled with X support' 13:49
like, forget that it's perl5, how do you depend on 'libperl.so with multiplicity' for example
nine Oh yes, like I need libperl but with -fPIC
From a distro perspective, it's probably fine to be able to handle only 90 % fully automatically as long as it's possible to detect the remaining 10 % automatically. 13:51
Like detecting that this dist runs custom code during configure, so we need to have a look if it's checking dependencies there. 13:52
mst well, yes, that's the dynamic_config flag from perl5 13:53
if that's set to 0, you don'#t need to run the Makefile.PL/Build.PL at all, you can just install the files from lib/ directly
of course about three dists on the whole of cpan actually do that 13:54
moritz an 80% could use it
nine So that's something that didn't work out that well :) But it didn't do so because it was probably introduced _after_ CPAN had a 5 digit number of dists. 13:55
mst actually, in part, it didn't do so because it was introduced by Module::Build which was such a fucking shitshow none of the installer writers wanted to support it 13:56
nine Of course we can fix that by making custom code opt-in instead of opt-out ;)
mst that's what Module::Build tried to do
resulting in lots of dists that should've opted in and didn't 13:57
resulting in lots of completely broken shit unless installers ignored the flag
that's why it got defeined to "it defaults to 1 unless explicitly present as 0"
nine Sounds like installers actually ignored the flag so people weren't forced to set it and so didn't? 13:58
mst that's not what I said
and then Module::Build::Tiny has been shaking out ExtUtils::InstallPaths 13:59
nine Anyway I'm thinking more like making this part of the Distribution::Builder interface. So one can ask the Builder if the dist is using custom code. The builder will know in what ways it supports such code and give a correct answer without the module author having to update redundant information.
mst which should make it possible to have a dynamic_config 0 installer
yeah. in our case, 'not setting the builder_class in META6.json' will be the indicator
also we're not trying to retrofit it to the same extent 14:00
also we're not hamstrung by being written by an apps dev with no clue about systems like Module::Build was
nine Exactly. We're starting in such a better place, that I'm just arguing for setting our goals high. 14:01
mst sure. just I'm arguing for "not painting ourselves into as many corners" being a high goal :D
nine The one major advantage is that we have had dependency handling in our installers right from the start which gave us dependency information in the META data instead of code. That also means that a module author may rely on the installer to deal with configure dependencies and won't have to copy installer code into inc/ 14:03
Meaning we can fix mistakes in Builder/Configurer classes centrally 14:04
[Coke] mst: btw, you joked in a slide about breaking the build for people with spaces - HA, jokes on you, that never worked. 15:55
(there's a ticket for it somewhere in RT)
mst [Coke]++ # can't. breathe. laughing. too. hard.
[Coke] also, toolchain might be interested in perl6.fail/t/BUILD
16:21 leont_ joined 16:26 Woodi joined
Woodi hi :) 16:26
mst .o/
Woodi gist.github.com/slunski/c722cf2c1d...576853537b ? 16:27
16:34 woolfy joined, woolfy left
nine Woodi: what exactly is that about? 16:40
Woodi nine: installation of modules me thinks... 16:42
nine Let me rephrase: what's the intention behind posting it? :) 16:43
Woodi eg. making installation "protocol" as simple as "cp -r lib/* ..." ? ok. intro: we have few kinds of repso, why don't have two kinds of modules: simple and not-simple :) 16:45
nine What would be the advantage? 16:46
Woodi nine, but you read this already ? or just asking first ? 16:47
nine I've read it
Woodi nine: so adventage is: to have plain repo that can be used by distros for 80% of modules 16:48
nine The only advantage I can see mentioned there is no hashes in file names? Which isn't much as we already have a plan for that anyway 16:49
Woodi can I read about this somewhere ? 16:50
nine What this proposal is missing is a description of how rakudo would look up a module with a dependency specification like use Foo:ver(v1.5..v2.5):auth(/.*\\@mycompany.com/)
mst basically, <hash>/Foo/Bar.pm
so one dist-level hash
[Coke] btw, I'm using a perl6 app in my toolchain here at work.
mst then you can have normal filenames, except where somebody uses utf8 in filenames
Woodi why not: readdir $repodir/mod/name/auth/ ? 16:51
nine Woodi: how does that work with use Foo:ver(v1.5..v2.5):auth(/.*\\@mycompany.com/)? 16:52
Woodi ls $rd/m/n/mycompany.com/* | awk v1.5..v2.5 ? 16:53
nine Woodi: see this talk for a lot of the reasons for the current structure: niner.name/talks/A%20look%20behind%...rl%206.pdf 16:54
Woodi: ls | awk would become slower with every module you install
Woodi: right now it doesn't matter if you've installed 5 or 500 modules (and versions), it will always take the same time to load a module 16:55
Woodi just ppl will not start using Perl6 becouse of sha's mess... 16:58
mst Woodi: how won't <hash>/Foo/Bar.pm fid that?
we KNOW the sha thing needs fixing
your proposal throws away all the parts that *do* work
17:03 domidumont joined
Woodi mst: why don't do simplest possible thing and have it usable ? nine++ .pdf shows problems with eg. x86_64-linux-threaded which is precompilation problem. pure code in $repodir do not have this and works for all possible v6 certified compilers; second problem is unicode but it is just choosing fs and mounting it, no need to resolve it via coding 17:03
mst Woodi: we are working towards a simplest possible thing 17:04
your proposal is not simpler, merely broken
Woodi and I especially take care to write clearly that it is just only one of possibilities. if you realy have one better on every scale then I do not make troubles :) 17:06
mst well I keep telling you what the better plan is
and you keep ignoring me
and going on about your non-working plan
Woodi mst: what is broken ? pls note that I am aware of v5 pitfalls
mst ...
I have work to do. you have fun. 17:07
Woodi ok, I will go too. cya :) 17:10
17:10 Woodi left 17:47 domidumont joined 18:24 domidumont joined 21:13 leont_ joined

Powered by Cro and the Raku® Programming Language.

Please report any issues / comments / feature requests as an issue on App::Raku::Log.

Thank you!