If you're interested in keeping up to date with the project I describe below, please follow me on twitter @NirvanaCore.
I had many of the same concerns with node.js. Every time I attempted to wrap my head around how I'd write the code I needed to write, it seemed like node was making it more complicated. Since I learned erlang several years ago, and first started thinking about parallel programming a couple decades ago, this seemed backwards to me. Why do event driven programming, when erlang is tried and true and battle tested?
The reason is, there isn't something like node.js for erlang, and so I set out to fix that.
For about a year I've been thinking about design, and for a couple months I've been implementing a new web application platform that I'm calling Nirvana. (Sorry if that sounds pretentious. It's my personal name- I've been storing up over a decades worth of requirements for my "ideal" web framework.)
Nirvana is made up of an embarrassingly small amount of code. It allows you to build web apps and services in coffeescript (or javascript) and have them execute in parallel in erlang, without having to worry too much about the issues of parallel programming.
It makes use of some great open source projects (which do all the heavy lifting): Webmachine, erlang_js and Riak. I plan to ship it with some appropriate server side javascript and coffee script libraries built in.
Some advantages of this approach: (from my perspective)
1) Your code lives in Riak. This means rather than deploying your app to a fleet of servers, you push your changes to a database.
2) All of the I/O actions your code might do are handled in parallel. For instance, to render a page, you might need to pull several records from the database, and then based on them, generate a couple map/reduce queries, and then maybe process the results from the queries, and finally you want to render the results in a template. The record fetches happen in parallel automagically in erlang, as do the map/reduce queries, and components defined for your page (such as client js files, or css files you want to include) are fetched in parallel as well.
3) We've adopted Riak's "No Operations Department" approach to scalability. That is to say, every node of Nirvana is identical, running the same software stack. To add capacity, you simply spin up a new node. All of your applications are immediately ready to be hosted on that node, because they live in the database.
4) Caching is built in, you don't have to worry about it. It is pretty slick- or I think it will be pretty slick-- because Basho did all the heavy lifting already in Riak. We use a Riak in-memory backend, recently accessed data is stored in RAM on one of the nodes. This means each machine you add to your cluster increases the total amount of cache RAM available.
5) There's a rudimentary sessions system built in, and built in authentication and user accounts seem eminently doable, though not at first release. Also templating, though use any js you want if you don't like the default.
So, say, you're writing a blog. You write a couple handlers, one for reading an article, one for getting a list of articles and one for writing an article. You tie them to /, /blog/article-id, and /post. For each of these handlers, any session information is present in the context of your code.
To get the list of articles, you just run the query, format the results as you like with your template preference and emit the html. If it is a common query, you just set a "freshness" on it, and it will be cached for that long. (EG: IF you post new articles once a week, you could set the freshness to an hour and it would pull results from the cache, only doing the actual query once an hour.)
To display a particular article, run a query for the article id from the URL (which is extracted for you) and, again this can be cached. For posting, you can check the session to see if the person is authorized, or the header (using cookies) and push the text into a new record, or update an existing record. Basically this is like most other frameworks, only your queries are handled in parallel.
The goal is to allow rapid development of apps, easy code re-use, and easy, built-in scalability, without having to think much about scalability, or have an ops department.
This is the very first time I've publicly talked about the project. I think that I'm doing something genuinely new, and genuinely worth doing, but its possible I've overlooked something important, or otherwise embarrassed myself. I don't mean to hijack this thread, but felt that I needed to out my project sometime. A real announcement will come when I ship.
If you're interested in keeping up to date with the project I describe above, please follow me on twitter @NirvanaCore.
EDIT TO ADD:
-- This uses Riak as the database with data persisted to disk in BitCask. The Caching is done by a parallel backend in Riak (Riak supports multiple simultaneous backends) which lives in RAM. So, the RAM works as a cache but the data is persisted to disk.
You know what I'm going to say, right? This is HN, we have but one great commandment, which we repeat like parrots!
Ship it tomorrow! ;)
Yes, you have overlooked something important, there will be something to be embarrassed by -- whether it turns up next week or next decade -- and we'll all have a good laugh. Don't sweat it. And don't worry that the thing isn't finished; the kind of geeks who might sign on at this stage like unfinished things; that is why they can't resist reinventing the wheel. Plus, it doesn't have to be finished to give people ideas, which is half the point. You are ready to start spreading the news; your writeup says as much.
The public repository beckons!
(Frankly, this sounds like a great experiment, although I would never be too quick to predict the end of the ops department. ;)
I agree completely. I just need to close the loop, so that people who use it are inspired to add to it, rather than frustrated trying to understand it. Once I can provide a trivial example of using it that makes sense, I'll ship. I can promise you I'm not holding out for some ideal!
I also shouldn't predict the end of the ops department, until I've had it running in production with a significant number of users.
I think it would be better to say- my goal is to have the ops department working on really interesting stuff, rather than shepherding a fleet of servers, every one of which has a different configuration.
Regarding 1), it is a great way to do things (never copy files around), but just make sure you have good release tools built around the db records. The approach that we use is that when things are published they automatically have a 'dev' release record for that version for the user that published it (so they can test) and individual versions have to be promoted ultimately to 'prod' for 'everyone'. The version you see is basically the most recent version released to your context (basically you and/or what machine you are on). The key is having great tools built around this process before something blows up and there is an emergency.
This is going to be an opportunity for the community. :-)
I have some plans in this area, but I couldn't guess how to best fit into other people's workflows.
What might be nice is if there was a way to sync a git repository with Riak, and then Nirvana could just pull the relevant code from that. Seems like it would be the best solution, but looking into that- from looking at possibly integrating with a github API (do they have one?) to command line scripts is something I'm punting on to focus on the essentials.
Again, essentially what we do in concept. The dev side is a separate db (git could represent this), and once something is published past dev the record is copied to the production db (Riak in your case). The dbs have in-memory local machine cached versions, for speed, but also for reliability. You don't want your N machines depending on one db point of failure, so the local cached copies exist to mitigate that. (They receive updates via a multicast mechanism.)
That looks like an interesting thing, but it is a something for node.js, not a "node.js for erlang".
By which, I meant to say "platform for building server applications in javascript, backed by the power of the erlang OTP platform."
Node.js gives server side javascript a platform, that's great. What I'm working on is giving server side coffeescript and javascript access to the erlang platform (and some really great erlang technologies.)
If you're not feeling too proprietary, hit me up on twitter. Might be some areas where we can collaborate. Either way, good luck with your project as well. I tried to not do this myself, I really did. :-)
I don't use Twitter much, but I'll definitely take you up on that when I have something a little more developed. I'm coming at this from the exact opposite direction as you (what I'm doing is closer to porting Erlang to Node than vice versa), but we're obviously thinking along the same lines— I'd love to compare notes.
I had many of the same concerns with node.js. Every time I attempted to wrap my head around how I'd write the code I needed to write, it seemed like node was making it more complicated. Since I learned erlang several years ago, and first started thinking about parallel programming a couple decades ago, this seemed backwards to me. Why do event driven programming, when erlang is tried and true and battle tested?
The reason is, there isn't something like node.js for erlang, and so I set out to fix that.
For about a year I've been thinking about design, and for a couple months I've been implementing a new web application platform that I'm calling Nirvana. (Sorry if that sounds pretentious. It's my personal name- I've been storing up over a decades worth of requirements for my "ideal" web framework.)
Nirvana is made up of an embarrassingly small amount of code. It allows you to build web apps and services in coffeescript (or javascript) and have them execute in parallel in erlang, without having to worry too much about the issues of parallel programming.
It makes use of some great open source projects (which do all the heavy lifting): Webmachine, erlang_js and Riak. I plan to ship it with some appropriate server side javascript and coffee script libraries built in.
Some advantages of this approach: (from my perspective)
1) Your code lives in Riak. This means rather than deploying your app to a fleet of servers, you push your changes to a database.
2) All of the I/O actions your code might do are handled in parallel. For instance, to render a page, you might need to pull several records from the database, and then based on them, generate a couple map/reduce queries, and then maybe process the results from the queries, and finally you want to render the results in a template. The record fetches happen in parallel automagically in erlang, as do the map/reduce queries, and components defined for your page (such as client js files, or css files you want to include) are fetched in parallel as well.
3) We've adopted Riak's "No Operations Department" approach to scalability. That is to say, every node of Nirvana is identical, running the same software stack. To add capacity, you simply spin up a new node. All of your applications are immediately ready to be hosted on that node, because they live in the database.
4) Caching is built in, you don't have to worry about it. It is pretty slick- or I think it will be pretty slick-- because Basho did all the heavy lifting already in Riak. We use a Riak in-memory backend, recently accessed data is stored in RAM on one of the nodes. This means each machine you add to your cluster increases the total amount of cache RAM available.
5) There's a rudimentary sessions system built in, and built in authentication and user accounts seem eminently doable, though not at first release. Also templating, though use any js you want if you don't like the default.
So, say, you're writing a blog. You write a couple handlers, one for reading an article, one for getting a list of articles and one for writing an article. You tie them to /, /blog/article-id, and /post. For each of these handlers, any session information is present in the context of your code.
To get the list of articles, you just run the query, format the results as you like with your template preference and emit the html. If it is a common query, you just set a "freshness" on it, and it will be cached for that long. (EG: IF you post new articles once a week, you could set the freshness to an hour and it would pull results from the cache, only doing the actual query once an hour.)
To display a particular article, run a query for the article id from the URL (which is extracted for you) and, again this can be cached. For posting, you can check the session to see if the person is authorized, or the header (using cookies) and push the text into a new record, or update an existing record. Basically this is like most other frameworks, only your queries are handled in parallel.
The goal is to allow rapid development of apps, easy code re-use, and easy, built-in scalability, without having to think much about scalability, or have an ops department.
This is the very first time I've publicly talked about the project. I think that I'm doing something genuinely new, and genuinely worth doing, but its possible I've overlooked something important, or otherwise embarrassed myself. I don't mean to hijack this thread, but felt that I needed to out my project sometime. A real announcement will come when I ship.
If you're interested in keeping up to date with the project I describe above, please follow me on twitter @NirvanaCore.
EDIT TO ADD: -- This uses Riak as the database with data persisted to disk in BitCask. The Caching is done by a parallel backend in Riak (Riak supports multiple simultaneous backends) which lives in RAM. So, the RAM works as a cache but the data is persisted to disk.