> I haven't checked out the original paper, but the solution presented on the blog is a non-solution. It only works for ascii characters and words composed out of ascii.
As I understand it, that was the original spec, which both Knuth and McIlroy wrote to. I agree that it is limited as you say.
> The shell solution, built upon standard tools, can not be extended to work in an international context but a custom solution in Python (or even Pascal) quite concievably could.
As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr. The Python equivalent would be to use the built-in Unicode support. (If Pascal had that, you could do the same in Pascal.)
However, it's worth noting that by specifying the problem that way you still have the issue of how the input stream (which is going to be bytes) is encoded. Essentially, the original spec declared by fiat that the encoding was ASCII.
Also, btw, you can express non-English languages in ASCII (though certainly not as wide a variety as in Unicode); the program as written does assume that words are composed only of the 26 standard ASCII letters, but it could easily be extended to include the ASCII special characters. Another exercise for the reader. :-) Though if you're going to do this kind of extension, it might be better just to go the whole way and handle Unicode.
But your assertion was that this was an easy problem solved with just a few lines of shell code. That assertion is only true in an extremely limited context where the only language possible is English and the only character encoding is ascii. That's where the Unix tools shines because they are great at handling problems easily expressable as regular expressions. But it is not a very realistic example, nor a fair comparison.
> As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr.
You should try that and then blog about it. :) I've spent lots of time battling issues with ascii-centric libraries. My conclusion is that it is not easy at all which is not strange because tools like tr and sed were written decades before unicode support became a must have. I can't say that it is impossible to write a shell script to count words in a text written in Arabic script, but it doesn't seem easy.
> But your assertion was that this was an easy problem solved with just a few lines of shell code.
Where did I assert that?
> I can't say that it is impossible to write a shell script to count words in a text written in Arabic script, but it doesn't seem easy.
Is there a definition of what counts as a "letter" in Arabic script? That is, which Unicode code points correspond to letters? And does Arabic share the convention that a "word" is a sequence of letters delimited by non-letters?
If the answers to those questions are "yes", then the extension of the algorithm already presented to handle Arabic is straightforward; it's just a matter of substituting the Arabic definition of "letters" for the ASCII definition. (With, as I said in my previous comment, the additional issue of determining the encoding of the input and output.)
If some of the answers to the above questions are "no", then you have an issue with the problem specification, not with the algorithm you're going to use to solve it.
> As I understand it, that was the original spec, which both Knuth and McIlroy wrote to. I agree that it is limited as you say.
My new opinion is that great programmers will go beyond the spec and make their software handle Unicode in cases like this. It's the right thing to do. If the spec didn't require units tests does that mean you don't write unit tests? No, you do it anyway because it's the right way to do things. BTW I'm still working on being a great programmer myself.
As I understand it, that was the original spec, which both Knuth and McIlroy wrote to. I agree that it is limited as you say.
> The shell solution, built upon standard tools, can not be extended to work in an international context but a custom solution in Python (or even Pascal) quite concievably could.
As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr. The Python equivalent would be to use the built-in Unicode support. (If Pascal had that, you could do the same in Pascal.)
However, it's worth noting that by specifying the problem that way you still have the issue of how the input stream (which is going to be bytes) is encoded. Essentially, the original spec declared by fiat that the encoding was ASCII.
Also, btw, you can express non-English languages in ASCII (though certainly not as wide a variety as in Unicode); the program as written does assume that words are composed only of the 26 standard ASCII letters, but it could easily be extended to include the ASCII special characters. Another exercise for the reader. :-) Though if you're going to do this kind of extension, it might be better just to go the whole way and handle Unicode.