By Adam B. Levine: Monday, March 11th saw calamity strike at the core of the Bitcoin
system – over the last 3 years the number of users has increased
exponentially, and the value along with it. In the past we’ve seen
large public thefts, but Monday’s event was fundamentally different. Bitcoin is basically one big public distributed ledger (known as The
Blockchain) where the ownership of every Bitcoin ever created is
tracked, accounted for, and verified automatically by various
participants in the system. Transactions can be detected within a few
seconds, but merchants generally take the hour or so to gather 6
confirmations to ensure the payment cannot be reversed.
Other participants, upon receiving news of the ownership transfer (payment), check the payer’s account to ensure that they haven’t already spent them publicly. A confirmation is the return message saying “Yes, this all seems to be in order, the value is truly available to be transferred to the intended recipient and we’ve updated our ownership records to reflect that.”
With that background in mind, on Monday the blockchain split in two without warning.
Why’d it happen?
Call it growing pains – Berkeley Database (BDB) was used for versions of the Bitcoin software .7 and below and as luck would have it, will accept a maximum number of changes per block. The protocol’s self adjusting difficulty seeks to issue a block every 10 minutes. As Bitcoin usage has increased, so have the number of transactions contained in the average block.
So what happened?
On Monday, a single block went through the system containing 1700 transactions, and was rejected by the participants still running on .7 due to how many changes it made. Had everyone been running .7, the block would have been rejected and the transactions processed in (hopefully) smaller blocks.
Because an upgrade to .8 was rolled out several months ago that switched to a different database system without the same limitation, we found ourselves with two separate blockchains (ledgers) each disagreeing with the other’s interpretation of reality. Those running .7 saw too-large blocks as invalid, while to those running .8, everything was normal and the larger blocks were acceptable and built upon.
Cause and effect
An alert went out immediately (at 1:30am UTC) from the mostly volunteer open source development team letting merchants know to not accept Bitcoins for the next few hours while the system sorted itself out. To miners, they asked that everyone revert back to .7 so no incompatible blocks will be generated. The average user was unaffected.
Crisis isn’t good for anyone, so participants quickly fell in line wanting to resolve the issue as soon as possible. Within 8 hours things were back to normal; If you live in America, chances are pretty good you slept through the whole thing, including the 23% drop in price, which vanished by morning.
Schrödinger’s Bitcoin
The “Double Spend” is the bogeyman of digital currency world. Since Bitcoins are just data and data can be copied, what’s to protect a merchant from being paid with money already spent elsewhere? This is the function of the blockchain, which represents the network consensus of reality.
When that consensus broke down due to the fork, it created a state best represented by the Schrödinger’s Cat thought experiment – one in which a single entity simultaneously and paradoxically exists in two states, only one of which can eventually be real.
During the disruption, one large Double Spend actually occurred and was paid out in US Dollars. When the faulty blockchain was abandoned, the reality consensus settled on the backwards compatible ledger. In this case, payment processor OKPAY was left without the Bitcoins they thought they received, and were also still out the nearly $10,000 they exchanged for them.
This is the most valuable instance of a successful Double Spend in the Bitcoin ecosystem, which can only occur under the Schrödinger’s Bitcoin scenario described above.
Luckily this story has a happy ending. Both parties “did the right thing” and have refunded each other. But will things work so well next time? Will there be a next time, is this situation repeatable?
I asked Gavin Andresen, Lead Developer on the Bitcoin project about the risks moving forward:
Question: Are “Double Spends” possible under normal circumstances?
Gavin: Double-spend risk is not a “yes or no” — like most risks, it is “more risky” or “less risky”, down to “So little risk I won’t worry about it. One of the reasons I still tell people “bitcoin is an experiment” is because everybody is still figuring out where the edge-case risks are.
Q: So for the average user or merchant, how much risk is there in every-day use?
G: For the, average user or merchant delivering a physical product to customers: I’d say very close to zero risk after 3 confirmations. Exchanges or merchants that deliver “cash-like” high-value products or services have their own particular set of risks, and they need to be more careful. merchants that deliver services that have a zero marginal cost to them (e.g. “Subscribe to my e-magazine”) have even less risk than users or merchants delivering physical products.
And all of that is modified if you have some trust in the person you’re transacting with. I recently sold a couple thousand dollars worth of bitcoin to a friend who promised to send me a check in the mail, and I was happy to do that.
Q: In your worst case scenario, where you flat out do not trust the person but want to do business with them, what would you recommend?
G: For ultimate trust transacting a huge amount of value to somebody who I think will try to rip me off– 24 hours would be the “100% safe” time. It really is confirmations and not time, but unless something wacky is happening those two are equal after 6 confirmations or so.
I pick 24 hours because that would be 120+ confirmations. I can’t conceive of a situation where “we” (the shared consensus of the bitcoin network) would allow a 120-block fork to happen that double-spent
Q: Now that the issue has been resolved for a day, any lessons learned or things that will be improved on?
G: actually one really stupid/simple thing ‘we’ should have done long ago: make it easy for bitcoin services to get alerts sent to their email/SMS/etc. We’ll be concentrating on things like improving communication during crises, because no two crises are exactly alike.
Regarding specific tasks: implementing a -alertnotify= is very high on my TODO list. That’s easy to do, and is the
thing I’m kicking myself for not doing sooner
Q: What does that mean?
G: That means “if you get an alert, run this command to tell me about it” where “this command” is specified by the service operator — send email, send sms, whatever
Moving forward
History would have us believe Bitcoin scandal leads to big losses in the trading value of the currency, but this time was different. Despite the currency itself seeming at risk and the impossible momentarily made real, the price immediately bounced back up to near all-time highs.
Bitcoin may be small as currencies go, but the hourly chart below speaks volumes given that every 1K Bitcoins represents about $50,000 USD.
So here we are, less than 72 hours after the event, and we’re
basically back to where we started. The system is stable, the fork
abandoned and the consensus reality once again agreed upon. And all
this with a basically volunteer development team who has no real ability
to make anybody do anything, outside of telling them it’s a good idea.
Compare that to the ongoing crisis in currencies controlled by central banks, who have the unilateral authority not only to make all decisions related to their respective monetary policy, but also the ability to issue more bills at will, should the need arise.
Lacking such an option, Bitcoin is forced to deal with its problems and try as quickly as possible to return to a state of normality. So does it work? For now, the answer is a definitive “yes,” but since it was just created in 2009, only time will tell.
From where I’m sitting this was just about as good an outcome as anyone could hope for. No value was lost, even though the system prides itself on irreversible transactions (which once you’ve dealt with Paypal’s merchant Chargebacks, you begin to see the appeal of).
The development team isn’t hiding the cause of the recent issue and is instead using this as a reason to dig into the response with work being done on an internal, comprehensive post-mortem.
They’ve taken the whole thing as a learning opportunity and have pledged to implement a more agile communication-alert system so if this does happen again, users will be alerted and large users will take appropriate precautions.
I like to see people with power admit they are not omniscient, and in this world of centrally planned money it’s too rare a thing.
One thing is certain: Bitcoin isn’t suffering for lack of its own Bernanke, and that should give us all reason to hope.
Adam writes about new technology at Mind to Matter and forestry, photography and custom wood products at NapaWood. You can contact Adam here.
Edited by Madison Ruppert
Other participants, upon receiving news of the ownership transfer (payment), check the payer’s account to ensure that they haven’t already spent them publicly. A confirmation is the return message saying “Yes, this all seems to be in order, the value is truly available to be transferred to the intended recipient and we’ve updated our ownership records to reflect that.”
With that background in mind, on Monday the blockchain split in two without warning.
Call it growing pains – Berkeley Database (BDB) was used for versions of the Bitcoin software .7 and below and as luck would have it, will accept a maximum number of changes per block. The protocol’s self adjusting difficulty seeks to issue a block every 10 minutes. As Bitcoin usage has increased, so have the number of transactions contained in the average block.
So what happened?
On Monday, a single block went through the system containing 1700 transactions, and was rejected by the participants still running on .7 due to how many changes it made. Had everyone been running .7, the block would have been rejected and the transactions processed in (hopefully) smaller blocks.
Because an upgrade to .8 was rolled out several months ago that switched to a different database system without the same limitation, we found ourselves with two separate blockchains (ledgers) each disagreeing with the other’s interpretation of reality. Those running .7 saw too-large blocks as invalid, while to those running .8, everything was normal and the larger blocks were acceptable and built upon.
Cause and effect
An alert went out immediately (at 1:30am UTC) from the mostly volunteer open source development team letting merchants know to not accept Bitcoins for the next few hours while the system sorted itself out. To miners, they asked that everyone revert back to .7 so no incompatible blocks will be generated. The average user was unaffected.
Crisis isn’t good for anyone, so participants quickly fell in line wanting to resolve the issue as soon as possible. Within 8 hours things were back to normal; If you live in America, chances are pretty good you slept through the whole thing, including the 23% drop in price, which vanished by morning.
Schrödinger’s Bitcoin
The “Double Spend” is the bogeyman of digital currency world. Since Bitcoins are just data and data can be copied, what’s to protect a merchant from being paid with money already spent elsewhere? This is the function of the blockchain, which represents the network consensus of reality.
When that consensus broke down due to the fork, it created a state best represented by the Schrödinger’s Cat thought experiment – one in which a single entity simultaneously and paradoxically exists in two states, only one of which can eventually be real.
During the disruption, one large Double Spend actually occurred and was paid out in US Dollars. When the faulty blockchain was abandoned, the reality consensus settled on the backwards compatible ledger. In this case, payment processor OKPAY was left without the Bitcoins they thought they received, and were also still out the nearly $10,000 they exchanged for them.
This is the most valuable instance of a successful Double Spend in the Bitcoin ecosystem, which can only occur under the Schrödinger’s Bitcoin scenario described above.
Luckily this story has a happy ending. Both parties “did the right thing” and have refunded each other. But will things work so well next time? Will there be a next time, is this situation repeatable?
I asked Gavin Andresen, Lead Developer on the Bitcoin project about the risks moving forward:
Question: Are “Double Spends” possible under normal circumstances?
Gavin: Double-spend risk is not a “yes or no” — like most risks, it is “more risky” or “less risky”, down to “So little risk I won’t worry about it. One of the reasons I still tell people “bitcoin is an experiment” is because everybody is still figuring out where the edge-case risks are.
Q: So for the average user or merchant, how much risk is there in every-day use?
G: For the, average user or merchant delivering a physical product to customers: I’d say very close to zero risk after 3 confirmations. Exchanges or merchants that deliver “cash-like” high-value products or services have their own particular set of risks, and they need to be more careful. merchants that deliver services that have a zero marginal cost to them (e.g. “Subscribe to my e-magazine”) have even less risk than users or merchants delivering physical products.
And all of that is modified if you have some trust in the person you’re transacting with. I recently sold a couple thousand dollars worth of bitcoin to a friend who promised to send me a check in the mail, and I was happy to do that.
Q: In your worst case scenario, where you flat out do not trust the person but want to do business with them, what would you recommend?
G: For ultimate trust transacting a huge amount of value to somebody who I think will try to rip me off– 24 hours would be the “100% safe” time. It really is confirmations and not time, but unless something wacky is happening those two are equal after 6 confirmations or so.
I pick 24 hours because that would be 120+ confirmations. I can’t conceive of a situation where “we” (the shared consensus of the bitcoin network) would allow a 120-block fork to happen that double-spent
Q: Now that the issue has been resolved for a day, any lessons learned or things that will be improved on?
G: actually one really stupid/simple thing ‘we’ should have done long ago: make it easy for bitcoin services to get alerts sent to their email/SMS/etc. We’ll be concentrating on things like improving communication during crises, because no two crises are exactly alike.
Regarding specific tasks: implementing a -alertnotify=
Q: What does that mean?
G: That means “if you get an alert, run this command to tell me about it” where “this command” is specified by the service operator — send email, send sms, whatever
Moving forward
History would have us believe Bitcoin scandal leads to big losses in the trading value of the currency, but this time was different. Despite the currency itself seeming at risk and the impossible momentarily made real, the price immediately bounced back up to near all-time highs.
Bitcoin may be small as currencies go, but the hourly chart below speaks volumes given that every 1K Bitcoins represents about $50,000 USD.
Compare that to the ongoing crisis in currencies controlled by central banks, who have the unilateral authority not only to make all decisions related to their respective monetary policy, but also the ability to issue more bills at will, should the need arise.
Lacking such an option, Bitcoin is forced to deal with its problems and try as quickly as possible to return to a state of normality. So does it work? For now, the answer is a definitive “yes,” but since it was just created in 2009, only time will tell.
From where I’m sitting this was just about as good an outcome as anyone could hope for. No value was lost, even though the system prides itself on irreversible transactions (which once you’ve dealt with Paypal’s merchant Chargebacks, you begin to see the appeal of).
The development team isn’t hiding the cause of the recent issue and is instead using this as a reason to dig into the response with work being done on an internal, comprehensive post-mortem.
They’ve taken the whole thing as a learning opportunity and have pledged to implement a more agile communication-alert system so if this does happen again, users will be alerted and large users will take appropriate precautions.
I like to see people with power admit they are not omniscient, and in this world of centrally planned money it’s too rare a thing.
One thing is certain: Bitcoin isn’t suffering for lack of its own Bernanke, and that should give us all reason to hope.
Adam writes about new technology at Mind to Matter and forestry, photography and custom wood products at NapaWood. You can contact Adam here.
Edited by Madison Ruppert
No comments:
Post a Comment