> Features are not better than reliable security.

Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all.

Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk?

No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend.

If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death.

Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures.

Things are not so black and white as to say "security is always the most important thing"

I did not say >"security is always the most important thing". I did not suggest that anyone should develop such that >" the thing you are creating has no features". I certainly never suggested that no one get out of bed because they might slip in the bath. These are strawman arguments.

You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems

I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down.

Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people.

Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8.

You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week.

You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features.

These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc.

Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring.

This is what you do if you are serious about risk.

Ok, I never said that the 'analysis' should be purely based on dollar value. I never even said it should be a mathematical model. You accuse me of a strawman and then turn around and do the same to me.

I was simply pointing out that we can never get to zero risk, and since we can't, we have to weigh risks based on consequences and probability.

> 1) fully examine the system for potential critical failure points/modes

Sure, to the best of your ability. How can you know for certain you have found all potential critical failure points? You can get pretty sure, but never fully sure. We still have industrial accidents, in every single industry in the world.

You also have to define what a 'critical risk' is. I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.

So your model isn't even mathematical, it's what, just a SWAG of the combined hazards and odds? That works for linear, small risks.

It absolutely does NOT work for serious risks, e.g., of death, serious injury, massive privacy violation, and other potential life-changing events.

The concept you are clearly avoiding or missing non-linear risk.

You (and amzn_engineer1) are advocating for simply subsuming risk assessment into the ordinary development cycle, and calling it "taking it seriously".

That is fooling yourself.

Taking it seriously is actually making full and serious effort OUT OF THE NORMAL DEVELOPMENT CYCLE for no other purpose than to SEEK and identify potential critical risks.

It is then engineering a variety of in-depth solutions to prevent those critical failure points from ever seeing the light of day. And implementing them. and testing them. And monitoring them.

>> I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.

This is an exact example of this sort of failure: 'it's not a priori bad'..., minimize it and streamline it into dev.

I really want to know in what world any sane person would say that it's OK to randomly divulging an intimate conversation to a contact or random recipient -- seriously, who would say that?

I mean sure, most conversations are benign, but some could be utterly life-changing if revealed. and that's OK with you?

It's worth observing that, firstly, aircraft still crash and bridges still collapse, amd secondly, that a culture of blameless analysis has gone a long way towards making them crash and collapse less.

Absolutely!

But note the blameless analysis is NOT the same as saying "meh" errors happen.

It is a culture and deliberate practice of allocating resources to seeking and classifying risks, and designing, implementing, and testing engineering and procedural mitigation strategies (vs. development as usual).