There of course is reversible matmul for reversible matrix. There is no reversible relu though.
But in any case I don't understand the claim of the article. If you can reverse the computation(say only use reversible matrix) you can do it for less energy?
demonstrated is the key word here. you can make most circuits ~reversible by just shuffling off to a pool of bits that you destroy eventually (but i presume you destroy a minimal amount), but is the juice worth the squeeze? i would worry that the scaling factor for floorplan is not linear in the matrix size (which is already O(nxk))
if you're also batching matmuls isnt there an unavoidable information loss that happens when you evict the old data and drop in the new batch?
You're mistaking invertible matrices for reversible computing, which are two unrelated concepts*. You can devise a reversible implementation of ReLU (and of anything else for that matter), using ansible variables. Like in addition:
add(x, a) = (x + a, x - a), and add†(y, b) = ((y + b) / 2, (y - b) / 2)
It's a well known thing in thermodynamics that a reversible process doesn't increase entropy (heat). So in theory, a reversible computer consumes no power whatsoever, as it doesn't leak any. In practice, most power leakage in real life computers is due to wire resistance, not the irreversibility of computations. Also, even with a perfectly reversible CPU, you would need to expend some energy to (re)initialize the state of your computer (input) and to copy its results out (output). Alternatively, once a computation is done, you can always "uncompute" it to get back to the initial state without using any power, at the cost of time.
If you want an example of a real invertible computer, look into quantum computers, which are adiabatic (reversible) by necessity, in accordance with the laws of quantum physics.
* Actually, you can represent reversible gates with invertible matrices, and that has quite profound implications. A gate/operation is reversible if and only if its corresponding matrix is invertible. But let's not get into that here.
Your g is not reversible though, its input space must be the same dimension than its output space. On the other hand, you're correct in that any irreversible function can be extended to a reversible one, although the process isn't always straightforward. The general way is to do somthing like:
f: V -> V
g: V x A -> V x A
with g(x, a) = g(f(x), b) for some value(s) of a. And b cannot be set to x, because then you can't find a back with g', and your function is not invertible.
IMO it would be more accurate to say: the delete operation is the one that we know thermodynamics and information theory must charge us for, on a physics level.
In current computers, we’re nowhere near those limits anyway. Reversible computing is interesting research for the future.
Quantum dot cellular automata looked like a good candidate for reversible computers in some post-cmos future, last time I looked at that stuff (years ago—so, it is probably time for a check-in). Notably, they do classical computing, they just exploit quantum effects for the logic gates.
I guess it would depend on the physical design of the compute elements (reversible computing is generally associated with post-CMOS tech). But, cutting power doesn’t necessarily reset a machine in general… think of a mechanical computer, cutting power removes the ability to change state.
Idk how accurate it is by my mental model of this would be that electricity is like a liquid and when you flip the switch, it drains out of the components creating heat through friction.
A fully reversible computers would consume no power whatsoever, so wouldn't require power to function. Instead, you would need power to (re)initialize its state, or to copy the results of its computations to e.g. your monitor. An unplugged reversible computer would be free to compute (and uncompute) its bits perpetually, unconcerned about the rest of the world. The functional programmer's dream, in a sense.
Thanks for that and the preceding comments. I was thinking about the possible paradox of getting a reset for free. But, the cost invariably comes at some point, e.g. when you restart and need to reset your state.
“Cutting power” is a thing because current computers require a periodic (and rather frequent) refresh to maintain the state of the system (primarily, but not exclusively, RAM). And indeed, one useful tactic to maintain that state when you need to cut the power for some reason is to supercool the ram so that it doesn't dissipate its charge as fast, basically making the system approach that ideal world.
i don't think that's categorically true. if your footprint from expanding out a circuit to make it reversible gets sufficiently large other things like resistance can dominate.
Other things like resistance already dominate (and always have). Reversible computing is the result of exploring the thermodynamic/information theory limits of what computation must cost.
In current chips we just charge and dump a bunch of parasitic capacitances every clock cycle.
People have been working on commercialising this stuff for decades, actually. The problem is that you are competing with the staggering level of investment that goes into CMOS. To get early revenue you need a niche which is forgiving of not being as good as CMOS in other dimensions - AI is almost certainly not that.
But in any case I don't understand the claim of the article. If you can reverse the computation(say only use reversible matrix) you can do it for less energy?