It’s a good idea, but a completely wrong solution. If someone is going to implement this, the proper way is not creating a PAUSE and CPAN mirror or anything like it. Corporate requirements are different. What you really need is a revival of http://debian.pkgs.cpan.org/ and also complete repositories for RHEL5, Debian, Ubuntu LTS, SLES9/10 etc. Those can be mirrored at will by usual and well-tested tools.

Thank you.

Update: Removed last sentence, it’s been a “thinko” on my side :(

I do consider PBP a good collection of recommendations. But sometimes, I loathe the impact it has on the community. It’s the Perl bible, Part II (just after the Camel book) and many people just go on believing in it. What we get as a result is a free community’s dependency on a non-free book.

At my place of work, we run Perl::Critic on SVN commits with PBP rule set. Everytime it finds something, it tells me to “Look at page XX of PBP for details”. So much for online help… Yes, I do have a copy of PBP in the office, even on my table. Sadly, it’s a german translation which has slightly different page numbering.

Why does a code analyzer cite some particular book edition? Imagine a second edition of PBP coming out. What do we get then, a command line parameter for book version? For a translation thereof? How many modules will we have to update? Usually a book gets updated when code changes, not other way around.

An even more heretical question: what happens if I don’t actually own a copy of PBP? Am I doomed to stay ignorant of best practices just because I’m just starting to learn and can’t or don’t want to shell out money for a book?

Other language communities are different. Both Ruby and Python give you extensive online documentation and also some dead-tree docs if you need them. But you don’t have to buy a book just to learn some best practices, those are readily available in blogs, wikis and what not. Perl’s community seems to trust in holy cows (camels or dogs for that matter) and just keeps insisting on buying books. “Modern” is something very different, though.

I know I can read PBP on Google Books, with several dozens of invisible pages. But PBP should have been online a long time ago. It should have been a community work from the beginning, since best practices is one of the first things a newbie needs.

There is only one way for PBP for the future: O’Reilly and Damian should open it up, just like “Higher Order Perl” has done. Make it downloadable at first, make it a community-driven project later. O’Reilly could even release a dead-tree edition every now and then, but the first step would be to free Damian from all the “please update PBP” e-mails — it’s perfectly probable that he doesn’t have time to do so and even more probable, no personal interest in bringing the second edition out. In that case, maybe he should raise his voice and work with O’Reilly on making that vastly important book a community project.

Yes, I know, CPAN is great. I even agree. CPAN is great because of the sheer amount of data collected. But it’s a complete disaster otherwise. I might be a bloody newbie in Perl world, but everytime I’m confronted with CPAN, I’m lost and confused — and there is a major flaw in CPAN causing that feeling: every module in CPAN is essentially an open-source project, but nothing at CPAN works under this assumption. It’s full of closed down silos.

Let’s start with a simple example: toying with CPANPLUS::Dist::RPM (or maybe it’s this link, who knows which is the canonical one) at work I’ve noticed it hangs sometimes, consuming 100% of CPU essentially doing nothing worthy. Let’s now assume I’d like to investigate this problem, but I don’t know if this is a bug or a mistake on my side.

So I go to the CPAN page of the package. Oh, there is a discussion forum, let’s click on that! Too bad, it’s broken. Bug reports? Oh yeah, there are whole three of them — none of which is my problem as far I can see. And I can actually barely see, since the visual component of that bug tracker makes Bugzilla of 1998 look good. But I still not sure that’s a bug, so I wouldn’t file one. What’s next? Maybe there is something new and relevant in development code in the revision control system? Oh wait, there isn’t any. CVS, SVN, Git, Mercurial, anything? Nope, no such thing on CPAN. Only release tarballs and some weird release differ tool. No revision control for an open source hosting in 2009, am I looking right?! Only way to ask something is to ask the author per e-mail? What about collaboration, patches, interactive community process for single modules?

Dear Github guys, if you happen to read this, please host the CPAN for us! Revision control, bug tracking, code review, documentation parser — if you could add some discussion forums, you’d be a perfect CPAN hoster!

So CPAN is so far: EPIC FAIL in discussion forums, somewhat FAIL in bug reports, EPIC FAIL in encouraging open development. Those are basic open source functionality nowadays, you know. And those are not nearly scratching the surface of critisicm.

Every other page on CPAN is different in design and interaction, there is no common and consice web interface, many different docs/search/rating mirrors which ultimatively produce a lot of Google spam. An awesome lot of cruft, a lot of broken modules which pop up prominently as first search result, no clear indication if a module is abandoned or actively developed. Even the most potentially useful features like dependencies’ resolution are crippled — dependencies work only in one direction, whenever I’d like to know how people actually use some module, I’m lost again. This is CPAN of today, confusing and rusty. CPAN is naked and it seems nobody wants to point that out. I do not want to think that nobody actually notices.

The situation with CPAN is symptomatic for the whole Perl community. Whether it’s Perl.com, Perl.org, Perl Mongers site, Perl Monks, use Perl or CPAN, it’s always the same: unreadable and misaligned content, incomprehensive navigation and straining colors, self-representation on the web coming straight from 1999 ¹. All the good code in the world and the power of the language won’t help anyone as long as people are alienated by ugly tools, visually and technically. Why can’t CPAN have the visual docs design from http://perldoc.perl.org, which at least features a syntax highlighter? Some CPAN mirror I land on every now and then from Google is even uglier than the one at http://search.cpan.org. Do we care at all about how those sites look? Do we care about fellow Perlers, about how hard they have to look for information? Why isn’t there some central site for Perl information? Why is every Perl project so independant that things like Perl Iron Man happen without cooperation with Planet Perl? ²

Perl community has so many possibilities but most of them stay unused. Most people are probably content with what they have and wouldn’t want to change anything. It’s fine, Perl’s way certainly supports that, but then we can forget about Perl revival. It’d be a shame, but we’d have only us to blame, not some superstitions about Perl being a “write-only language” or “ASCII soup”. The first impression counts and many newbies might not make it to the code at all — they’ll struggle with finding tutorials first. They won’t find out why Perl is great and will leave for other, probably inferior, languages, because they’ll be reading some ugly outdated quickstart documentation from 1997. They won’t find the shiny things, but they should be able to — as their first Google search result.

Disclaimer: I’m essentially talking about l10n, but most people know it as i18n, so I’m keeping “i18n” in text.

The i18n problem

When it comes to making your application tranlatable in Perl, there are actually two schools of doing this: via Maketext and via GNU gettext. GNU gettext is the most known software translation tool used in most open-source projects while Maketext is a child of the Perl world. And the bad thing is: Maketext is currently more popular, but if you are using Maketext for making your application translatable, you are doing it wrong!

Let’s look at how Maketext works, according to its documentation and contrast that with the gettext way.

Maketext manual defines the process as following (quoting freely):

• Decide what system you’ll use for lexicon keys (i.e. base language)
• Create a class for your localization project
• Create a class for the language your internal keys are in
• Go and write your program
• Once the program is otherwise done, and once its localization for the first language works right (via the data and methods in Projname::L10N::en_us), you can get together the data for translation.
• Submit all messages/phrases/etc. to translators
  • Translators may request clarification of the situation in which a particular phrase is found
  • Each translator should make clear what dependencies the number causes in the sentence
  • Remind the translators to consider the case where N is 0
  • Remember to ask your translators about numeral formatting in their language
  • The basic quant method that Locale::Maketext provides should be good for many languages. […] For the particularly problematic Slavic languages, what you may need is a method which you provide with the number, the citation form of the noun to quantify, and the case and gender that the sentence’s syntax projects onto that noun slot.
• Once you’ve localized your program/site/etc. for all desired languages, be sure to show the result (whether live, or via screenshots) to the translators.

There is a lot of sense in this and this has certainly been valid back in 1999, but a lot of work in this process is not specified. For example, the translation process itself is questionable:

• How do you “Submit all messages/phrases/etc. to translators”?
• How do you integrate translations back from translators?
• How do you resubmit translation strings if they change?
• How do you communicate “situation in which a particular phrase is found” (i.e. context)?
• What happens if one phrase has to be translated differently depending on context? How does one implement that in a module properly?
• How does the translator “make clear what dependencies the number causes”? At what extents does that happen? Will the developer even understand him at all?
• Does the programmer really have to understand all of implications of each language implemented? Should every programmer on the team understand them?
• Who actually implements that “quant” method? How? What about languages with expections?

One basic, but fatal, mistake Maketext does is off-loading a lot of linguistic work onto programmer.

• One particularly important point is the plural forms support (‘1 apple’, ‘2 apples’), which is important for many languages outside of USA and Western Europe . Maketext requires you to write a quant function that gets a string and a number as parameters and does some voodoo to produce the right string. Voodoo is undefined. In gettext it is — a formula for producing plural forms is defined which selects one of provided plural phrases.
• No translator in his sane mind will ever write a Perl module for a language (they aren’t programmers, remember?), the programmer will have to do it and will also have to understand the implications.
• The quant notation ("Your search matched [quant,_1,document]!") foolishly assumes word order is the same in all languages. Implementing a quant method properly would require passing the whole sentence into the function and doing a complete linguistic transformation which is highly non-trivial and better done by human.
• Most of those linguistic “conventions” like number formatting or plural forms do not change over time and can be compiled at one place. One such place is Unicode’s CLDR project, which also includes plural form building and number/date formatting among other country- and language-dependant data.
• It can’t even be assumed that the translators actually know all of these conventions! They might assume they know them, but translator is not necessarily doing translations for a living, he might be a volunteer, like in most open source projects. Imagine what happens when an amateur translator explains the inner workings of his native language to a programmer?

Compared to this gettext has a saner, more practical approach — they provide a standardized translation string format, handle updates of message catalogs cleanly, provide all necessary tools for message extraction, don’t require any additional modules, work mostly language-agnostic, provide contexts and translators’ comments, even plural forms calculation formulae are explicitely noted in the manual. It also emphasizes asynchronous translation: translation strings can be extracted and imported at any time in the lifecycle of a project. A developer essentially has to do the following:

• Implement using gettext in his project (depends on the language used)
• Mark extractable strings
• Run extraction and merging scripts (mostly included by gettext)
• Submit translation files to translators
• Copy received translations back into the project

gettext of course is not perfect. It lacks several vastly important features, like proper gender support (e.g. “He was born” and “She was born” is different in Russian). But it generally follows the “It mostly works” principle, making features needed 95% of the time available. Workflow tools make using gettext a snap. Compared to Maketext it is also easier to support for the programmer and easier for the translator to produce translations. The dreaded quant function actually makes using Maketext properly for translations impossible.

Apart from those techical shortcomings, there is a bigger threat.

Community separation

Remember TPJ13? TPJ13 is an excellent summary of i18n problems, which every developer, even non-Perl one, should read. It’s solution part is hopelessly out-of-date — don’t forget, TPJ13 is getting ten years old this year. Back in 1999 gettext hasn’t had any plural forms support and also lacked many other features so the authors’ point used to be valid at that point. However, gettext had implemented its support for plurals rather fast and at that time Maketext should have been retired immediately. Sadly, this has not happened.

That misunderstanding haunts us until this day. Every novice Perl hacker is introduced to TPJ13 and tends to believe Maketext is the way to go. Failing to see its shortcomings however, yields in well-meant but still failed creations like Locale::Maketext::Lexicon which tries hard to bring the world of gettext to Maketext-infected minds. What we get is crazy stuff like (verbatim from the POD)

#: Hello.pm:11
msgid "You have %quant(%1,piece) of mail."
msgstr "Sie haben %quant(%1,Poststueck,Poststuecken)."

instead of a proper (German spelling corrected a bit):

#: Hello.pm:11
msgid "You have 1 piece of mail."
msgid_plural "You have %d pieces of mail"
msgstr[0] "Sie haben 1 Poststueck"
msgstr[1] "Sie haben %d Poststuecke"

The former has virtually no tool support (not even gettext’s extraction routine xgettext), but extraction is supported by home-grown xgettext.pl (notice the .pl suffix). And there we have some fatal stuff going on:

• Locale::Maketext::Lexicon is considered the solution for using Maketext with .po files.
• Neither Locale::Maketext::Lexicon nor xgettext.pl have any notion of proper gettext plurals
• .po files created by xgettext.pl are not fully supported by translation tools like PoEdit, KBabel, Launchpad Rosetta, 99translations.com etc.
• Catalyst::Plugin::I18N, the only i18n plugin for the extremely popular Catalyst web framework, is based on Locale::Maketext::Lexicon
• xgettext.pl has support for Template-Toolkit templates, YAML, FormFu and Mason. Original gettext’s xgettext does not.

So there we have it: Perl hackers mostly use tools which are unsuitable and incompatible with the rest of the world without knowing it. The right tools actually can’t help them become “sane”, since xgettext can’t extract all those formats which xgettext.pl can and I don’t think that’ll change sometime soon.

Alternatives

Luckily, some hackers have produced a libintl-perl library which basically re-implements GNU gettext in Perl. There is a pure Perl implementation of message catalogs called Locale::gettext_pp, an XS version called Locale::gettext_xs (Warning: this one has some problems with mod_perl2!), a Perl wrapper around that (Locale::Messages) and building upon that an excellent Perl-y implementation of the framework Locale::TextDomain. These tools are worth your time.

Even though we have Locale::TextDomain, what should be done to amend the whole Maketext situation? I’d propose several possible actions:

• Read the GNU gettext Manual to fully understand what these tools can do for you
• Educate your colleagues, tell them about this article and explain the differences
• If you can, port your current code to Locale::TextDomain
• Don’t use Maketext for any new code
• Update important code using Maketext like the Catalyst plugin mentioned above to support gettext
• Update TPJ13 to reflect the situation
• Port extraction routines from xgettext.pl to xgettext

This and general awareness of the issue should bring Perl’s i18n back on track. Thank you for reading!

• Google’s blog converter
• Emacs Starter Kit

Let’s hope both produce something usable. And now I’m sort of relieved, since contributing to existing projects is simpler than starting new ones, especially with my limited knowledge on both subjects. Just like I’ve been shown by the second one, since I didn’t know about ELPA yet.

A quick introduction for those who are unfamiliar with cpan2dist: it fetches a Perl package from CPAN and installs it into your system using distribution’s tools. Thus you can resolve possible distribution upgrades painlessly, since every Perl package is also shown in your package management. It’s part of Perl 5.10 distribution and makes native package installation easy.

There are however some catches and the first hurdle comes at the very beginning: distribution-specific packaging is done with plugins and sadly Intrepid does not include the Debian plugin for cpan2dist, CPANPLUS::Dist::Deb. Since you can’t install that one from packages yet (due to lack of CPANPLUS::Dist::Deb, welcome to recursion), you’d have to install it manually — the good thing is, it will be the only package installed manually and it also can be replaced with a packaged one after bootstrapping the whole system. Just run

and the first step is done!

Now cpan2dist can be used. Let’s reinstall CPANPLUS::Dist::Deb, this time as a Debian package!

It should give you a sudo prompt at some point, this is because it installs the Debian package already, so you should be done in a couple of seconds.

The options’ meanings (in this order): the desired format of the packages, pre-requisites should be all built, packages should be installed, tests should not be executed before build (time saver) and module CPANPLUS::Dist::Deb is being built.

However, we are not done yet. You’d notice that cpan2dist tries to resolve some dependencies which are provided by basic Perl packages perl, perl-base and perl-modules. These might, but in most cases should not be downloaded and built from CPAN. Luckily, cpan2dist can be provided with an “ignore list”, the downside is that the default one is tiny und thus mostly useless. We’d have to recreate this list from our system:

This is my recipe. I get the list of all *.pm files from perl, perl-base and perl-modules and re-create the module name from its path. Every entry becomes a pattern (e.g. ^Test::Simple$) so that module names wouldn’t match as substrings and the whole list is dumped to ignorelist. Now we have to add an --ignorelist ignorelist option to the cpan2dist command line.

This should round it up — if you are lucky, everything goes well and you’d have an installed module as a Debian package. If you have some bad karma on a particular day, you’d end up with some errors, all of which can be solved with some manual package installation and careful reading.

One problem remains though: cpan2dist doesn’t seem to check whether a particular package is available from the archive. But that’s only a small nuisance in an otherwise really useful package.

However, not every website supports private links and there we have a problem: I’d like to use a couple of private feeds with Google Reader but can’t. Or better couldn’t, since I’ve made myself a late Christmas present: a Google Reader private feeds workaround!

It’s a small Perl script intended to be running somewhere on your own hosting. What does it do? Not much: given a GUID it fetches an associated website and returns it to the requester. If GUID is not found, nothing happens. If an error occurs, nothing is returned either. So simple. You still have to take care of the feed list though.

I will upload the code and some instructions to the newly created Google Code project shortly. Stay tuned!

I’ve been following ikiwiki, another site generator, for about half a year now. Actually, it’s a hacker’s dream, blogging with Emacs, storing the whole history in Git, having your site update automagically and being mostly secure because most web-visible content consists of generated static pages. For me, being statical is just a nice add-on, but I really really like the idea of blogging with my $EDITOR instead of some idiotic web-page form (but that’s a completely separate ranting topic). Lack thereof in Wordpress is the main reason I’m so lazy about blogging.

However, there is one major flaw, which so far has been exhibited by every RCS-enabled blogging engine I’ve seen: these tools violate MVC. Simple case goes like this: user creates a new blog entry somewhere under posts/2008/12/26/my-blog-post.mdwn, which gets transformed to /posts/2008/12/26/my-blog-post.html by the engine. There might be some derivation, but the principle stands — directory structure in the source is being mapped one-to-one to the web. However, this means that entry’s representation in a URL is directly attached to the content of this entry. Separating concerns, anyone?

Actually, it seems like correct behaviour: put a bunch of content into a storage and let some website generator do its work and present this content to the site visitor. In each blogging system each text is available at several locations: entry’s permalink, month’s overview, tag’s overview, category’s overview, etc. The problem is in our case that a single web-site location for the content is assumed and hard-coded by the author, every other is generated, which is to say the least, inconsistent.

A single blogger can play different roles on his website at the same time: designer, admin, coder, author, editor, usability expert etc. It’s important to remember which role starts when. Putting a text into the system is author’s task, but the editor’s the one to decide where to put it on the website. Depending on situation, there might be several locations, might be just one or even none if content is not to be published until a certain date. But it’s still in the database, it’s its visibility that changes.

Therefore separating concerns in this case requires web-site generation which relies only on content’s metadata and a set of publication rules. The contents of each page on the website could be defined by subroutines returning the entries. For example, if we assume that a list of all articles on a web-page is available (@posts), we could define (pseudo perl code):

• All posts: "/posts/" => map {$_.summary} sort {$a.date cmp $b.date} @posts
• All posts for a certain date: "/posts/\d{4}/\d{2}/\d{2}/" => map {$_.summary} grep {$_.date ~= "\1/\2/\3"} @posts
• An index page: "/" => grep {$_.category ~= "start" } @posts

In this case, it’s irrelevant how these pages are stored, the engine needs to make sure @posts is available and contains everything from a site.

In this context it’d make sense to define permalink strategy too. Taking an ID of an article as its permalink, one could define $permalink = sub { "/id/" . $_.id } to be a canonical generator for permalinks for each article.

What happens if permalinks change? Consider permalinks containing slugs (many people like them). Slugs are mostly auto-generated, so that permalink based off a slug will change if you update the title of your blog post. If it doesn’t, it might not be appropriate for the post anymore and thus misleading for the visitors. You could also change you permalink strategy completely, for whatever reason. If your writing is somehow relevant and linked to on the net, you’ll break a lot of links on other websites. Nobody wants that, but everyone does that every now and then.

If we however additionally save a URL and the subroutine which has been used for routing each time then we’d have a redirection permalink list. It could be checked by default if no current page definition matched. This way for each part of your web-site which has been visited at least once, you’d have a routing available and thus, these URLs would not die. This could help our permalinks be at least a little bit more permanent.

So to sum it up: in a perfect world we’d be putting our blog entries under version control in an arbitrary order, making sure our metadata is correct. A rule set would define how the website is built together based on this metadata. Permalinks would not die at the wave of a hand, defining additional views and URL patterns would be as simple as adding a single text line to a file. A couple of HTML templates would round up this whole stuff. A hacker’s ranting would be a little bit easier in a long run…

Geez, games… Computer games… I just don’t quite know what to write about them, since I haven’t been playing much in the last seven or eight years, I’ve willingly missed every single gaming fashion in the last decade. I’ve seen and played a bit of "Counter Strike", I’ve seen people playing "Need for Speed" or "Grand Theft Auto", I’ve talked long hours about psychological problems concerning "World of Warcraft" without having played or seen a minute of that, I’ve heard about Lineage and wondered what that might be until I’ve researched and found out it was a russian "WoW" competitor. I’ve been excited about and haven’t played a single minute of "S.T.A.L.K.E.R.", damn, even "DOOM" has been a game I’ve played three years too late. I’m a gaming failure and I admit it. Not that I’m ashamed of it…

It’s just that I find hacking my linux box, creating little scripts and reading a lot of articles and books on computing topics vastly more exciting that gaming. I’ve saved a lot of money by not buying the best graphic accelerator on the market, by not upgrading my workstation to the greatest, by not arguing about 1200dpi mice vs. 1500dpi mice with or without a battle mouse pad. I’m not an energy drink addict. I’m not into battle strategy, neither in "Counter Strike" nor in "Warcraft". I haven’t felt the need to imagine myself being a dwarf with a +8 power hammer. I’ve never been to the coma-drinking festivals called LARP. Sorry, I’m boring. I can spend the whole day and night reading about virtues of classical vs. prototype-based OO, my feed list in Google Reader is at 86 feeds at the moment, 814 entries unread — and most of them are really interesting. In fact, I’ve been going on like this since late 2000 and I’ve got every single job I’ve had since 2002 because of this. I love educating myself, so I’m doing this most of the time.

Coming back to the topic, I’m sometimes tempted to play something. Sadly, my games selection is somewhat limited by my OS, I’d have to reboot to play most of the games, so this is one more reason I’m not an active gamer. But still, sometimes I play and these are my top 3 games I’ve found entertaining recently and actually played long hours (apart from Free Cell, that is ;)):

1. Prince of Persia This one game — or better, three games since I’m talking about all three modern PoP games — is a masterpiece. The original "Prince of Persia" has taken a lot of time playing it, kids on the block talked about having saved the princess in 20 minutes, about how to kill the reflection prince, about Jafar etc. It was the game at that time, excellent in gameplay and graphics. When the second "Prince" came out, it was a kind of disappointment, as it wasn’t that much better. Still, we played it. The third one, "Prince of Persia 3D" was a extraordinary failure and still, I tried playing it. When "Sands of Time" came out, it was like a legend coming back, stronger than ever. I’ve been telling my friends that this should have been the original "Prince" if we had 3D graphics at that time. Excellent animation, intuitive controls, extraordinary fights, it had it all. Then there was "The Warrior Within", which was just as mindblowing — combining "The Matrix", "Mortal Kombat" and "Prince of Persia" has been pure genius. The same can be told about "The Two Thrones". The game was fun to play, it was difficult enough not to fall asleep, not too hard not to be turned off and long enough to have about two weeks of continuous fun.

2. Frets on Fire This is a new game on the block and it’s a free (as in speech) "Guitar Hero" clone. Being an extremely simple concept and providing no story at all, it’s still more fascinating than many other games currently on the market. Just play the guitar and enjoy it!

3. Pro Evolution Soccer Well, this one is pretty simple. I like soccer, I like playing it and I like playing it on my computer. Not that I was good at it, but I still can enjoy it. PES has a hard stand against EA-produced FIFA-branded game series, but it’s nevertheless a simulator with a better game feeling. I’ve actually bought a gamepad a long time ago to play it.

Apart from these, there are some games I still know from my childhood and which can take any number of hours of my free time. These are the Monkey Island series and Alley cat. Particulary the latter is a gem of arcade gameplay, being the only game known to man that looks great in CGA, running at normal speed even on current quad-cores despite its age (try that with Digger!) and providing a simple, but fascinating aim.

I also remember playing Moorhuhnjagd a lot when it first came out and also a great night spend at a friend’s place playing Mortal Kombat and a Tron clone that whole night. By the way, Armagetron is one great game, the best implementation of the tron-principle I’ve even seen.

That’s about it, there is not a lot more to say about my gaming preferences. Hope you had fun reading this stuff :)