F%$K Bad Research: Analyzing Flawed Bitcoin Mining Studies

This piece is part of Bitcoin Magazine’s “The Halving Issue”. Click here to obtain your copy. It also serves as report #1 of the “FUD Fighters” series powered by HIVE Digital Technologies LTD.

Forget Subpar Research: After a month of dissecting a bitcoin mining study, all I have is this emotional aftermath.

“We must acknowledge that our opponents have a significant edge in our debates. With just a few words, they can announce a half-truth, and we must resort to lengthy and tedious discourses to illustrate its incompleteness.” — Frédéric Bastiat, Economic Sophisms, First Series (1845)

“The energy expended to refute nonsense is significantly greater than that required to produce it.” — Williamson (2016) discussing Brandolini’s Law

For too long, we’ve had to deal with the repercussions of ineffective academic inquiry into bitcoin mining’s energy consumption and environmental repercussions. The fallout from this poor research has given rise to alarming headlines that have transformed earnest individuals into furious lawmakers and obsessed activists. To spare you from the pain of sifting through one of these careless papers, I’ve committed myself to the bitcoin mining deities and conducted an exhaustive analysis of a recent study from the United Nations University, published in the American Geophysical Union’s Earth’s Future. Only the most intrepid bitcoin enthusiasts should proceed to the next sections, while others may wish to return to monitoring market fluctuations.

Some may have gasped in disbelief at my bold assertion in the introduction that the most prominent studies on bitcoin mining are lacking in credibility. If you’ve read Jonathan Koomey’s 2018 blog about the Digiconomist—alias Alex deVries—as well as reports from Coincenter in 2019 or the works of Lei et al. in 2021, Sai and Vranken in 2023, and Masanet et al. in 2021, it’s clear that there’s an extensive body of literature reaffirming that energy modeling for bitcoin mining is in disarray—this issue is not confined to bitcoin alone. It’s a challenge that data center energy studies have been grappling with for years. Experts like Jonathan Koomey, Eric Masanet, Arman Shehabi, and the amicable Sai and Vranken (we’re still on a surname basis) have produced enough documentation to wallpaper at least one men’s restroom at every bitcoin conference that occurred last year, proving this point.

In my cherished sanctuary, which resides in my bedroom closet, I maintain an elegantly simplistic altar to Koomey, Masanet, and Shehabi, honoring their years of efforts to enhance data center energy modeling. These masters of computing have made it crystal clear: if your research lacks granular data and you depend on historical patterns while neglecting trends in IT equipment energy efficiency and the drivers of demand, then your work lacks rigor. Thus, with one sweeping yet meticulous move, I dismiss Mora et al. (2018), deVries (2018, 2019, 2020, 2021, 2022, and 2023), Stoll et al. (2019), Gallersdorfer et al. (2020), Chamanara et al. (2023), and all the others mentioned in Sai and Vranken’s exhaustive literature review. May these fallacies be consumed in one grand yet metaphorically magnificent inferno somewhere off the Pacific Northwest. Journalists and policymakers, I implore you to cease your consultations with Earthjustice, Sierra Club, and Greenpeace, as they are woefully misinformed. Forgive them their transgressions, for they are simply misguided. Amen.

With that, dear reader, I shall now recount an intriguing tale regarding a recent bitcoin energy study. I beseech the bitcoin gods that this will be my final exploration of this phenomenon, and the last one you’ll ever need to digest, but I suspect these gods are vengeful and will show me no leniency, even in a thriving market. One deep breath (cue Heath Ledger’s Joker) and away we go.

On a rather gloomy October afternoon, I was tagged on Twitter/X in a post referencing a new bitcoin energy study authored by some individuals connected to the United Nations University (Chamanara et al., 2023). I was blissfully unaware that this study would provoke my autistically intense focus, plunging me into a state reminiscent of a psychedelic experience, compelling me to concentrate on this paper over the following four weeks. While I may be exaggerating about the drug aspect, my memory of this period resembles a technicolor fever dream of a problematic relationship. Remember Frank from the critically acclaimed 2001 film, Donnie Darko? He was present, too.

As I began compiling notes on the study, I found the report by Chamanara et al. to be quite convoluted. The paper is puzzlingly constructed, relying heavily on the de Vries and Mora et al. studies. It employs the Cambridge Center for Alternative Finance (CCAF) Cambridge Bitcoin Energy Consumption Index (CBECI) data without acknowledging the model’s limitations (refer to Lei et al. 2021 and Sai and Vranken 2023 for an extensive critique of CBECI’s modeling issues). Furthermore, it mistakenly merges findings from the 2020-2021 period with the current landscape of bitcoin mining in 2022 and 2023. The authors also employed an environmental footprint methodology that suggests it might be feasible to manipulate a reservoir’s size based on one’s Netflix binge-watching habits. This inference aligns with Obringer et al. (2020), which the UN study cites as a methodological foundation. Interestingly, Koomey and Masanet have also criticized the methodology of Obringer et al. I’ll light another soy candle in their honor.

Here’s a clearer summary of the main issues with Chamanara et al. (by the way, their corresponding author has yet to respond to my request for their data so I could verify it, not merely trust it. 🥴):

The authors merged electricity usage data across multiple years, overstepping the conclusions that could be drawn from their methods.

The authors utilized historical trends to make present and future forecasts, despite a wealth of peer-reviewed literature clearly demonstrating that this leads to overestimations and hyperbolic claims.

The paper asserts that it will provide an energy assessment to reveal bitcoin’s true energy use and environmental impact. They employed two data sets from CBECI: i) total monthly energy consumption and ii) average hashrate share of the top ten bitcoin mining countries. It’s important to note that CBECI is based on IP addresses tracked across several mining pools. CBECI-affiliated pools constitute an average of 34.8% of the total network hashrate, leading to significant uncertainty in their data.

After about an hour of Troy Cross reassuring me as I stood on the edge of a metaphorical precipice, feeling a deep, unsettling dread that no amount of cognitive behavioral therapy could alleviate after realizing the troubling state of this study, I inferred that the equation used by the authors to calculate energy usage shares for each of the top ten hashrate-dominating countries (according to IP address estimates) was likely as follows:

Don’t let the math intimidate you. For example, suppose China has a hashrate share of 75% for January 2020. Now assume that the total energy consumption for January 2020 was 10 TWh (these are hypothetical figures for simplicity). For that month, China would have consumed 7.5 TWh of energy. Retain that number in your memory palace and perform the same operation for February 2020. Next, sum the January energy use with February’s energy consumption. Continue this process for each month until you have an aggregate for all 12 months, which will yield CBECI’s calculated annual energy consumption for China in 2020.

Before I present my findings in table form, let me clarify another important aspect of the UN study. This research employed an outdated version of CBECI data. To be fair, the authors submitted their paper for review before CBECI made amendments to their machine efficiency calculations. However, this means that Chamanara et al.’s findings are far from realistic, since it’s now believed that the older CBECI model overestimated energy consumption. Furthermore, my analysis was limited to data until August 31, 2023, as CBECI transitioned to the newer model afterward. CCAF graciously shared their older data with me upon request.

Another challenging aspect of this study is that it aggregates the energy consumption figures for both 2020 and 2021, which can be misleading; the prominent text states, “Total: 173.42 TWh.” Additionally, the figure caption says “2020-2021,” which for many would imply a 12-month span, not 24. To clarify, I separated the years so that everyone could see how those numbers were derived.

Take a look at the rightmost column labeled “Percent Change Between 2020 + 2021 Calculations (%)”. I computed the percent change between my figures and those of Chamanara et al. Curious, isn’t it? Based on discussions I’ve had with CCAF researchers, the numbers should align. Perhaps a smaller change isn’t reflected in the changelog, but our figures differ slightly nonetheless. The data CCAF provided shows a larger share for China and a smaller share for the United States compared to the UN study. Nevertheless, the overall totals are fairly similar. Therefore, let’s credit the authors with an adequate calculation given CBECI’s modeling limitations. Just remember that labeling their calculation as acceptable does not imply that employing historical estimates to make assertions regarding the present and future, and consequently shaping policy, is justifiable. It isn’t.

One evening, working by the dim glow of candles, I glanced to my left and saw Frank’s unsettling black eyes (the Donnie Darko character from earlier) gazing at me as two glistening pieces of coal amid pearly sand. He reminded me that this report was still unfinished, intertwining thoughts of time travel. I tugged at my soft curls (I switched to a bar shampoo, a miracle for frizz) and accepted my fate, as Willie Nelson’s 1974 Austin City Limits pilot episode echoed through the mono speakers of my cheap Chinese monitor, propelling through my ears like heroin coursing through Lou Reed’s extensive network of veins. Reluctantly, I acknowledged that I had to delve deeper into this research. I needed to analyze the 2020 and 2021 CBECI data more thoroughly to demonstrate how critical it is to conduct an annual analysis rather than amalgamating years into one figure. Realizing I was out of my preferred alcoholic beverage, I dished myself a splash of sherry in a Shirley Temple (shaken, not stirred) and took a swig of homemade antiseptic acquired during the lockdown.

I rummaged through my notes. I’ve collected ample notes because I take this seriously. What about the mining map discrepancies? Can we examine this through a two-year breakdown? What developments occurred for each of the ten listed countries? Could that offer insight into where hashrate migrated post-China ban? What about the crackdown in Kazakhstan? Although that falls after 2021, the UN study acts as if it never occurred when discussing the present state of mining distribution.

To the authors’ discredit, they neglected to inform their peer reviewers and readership that the mining map data only extends until January 2022. Thus, despite their assertions about the current energy mix of bitcoin mining, they are utterly mistaken. Their analysis merely reflects historical trends, not current or future realities.

Look at this colorful graph representing CBECI’s estimated daily energy usage (TWh) from January 2020 to August 31, 2023. At this broad scale, there’s considerable variability. Furthermore, it’s evident upon inspection that each year exhibits distinct energy use characteristics. Several factors could explain this variability: bitcoin price fluctuations, difficulty adjustments, and equipment efficiency, alongside broader regulatory influences, like the Chinese bitcoin mining ban instituted in 2021. A significant number of Chinese miners relocated to other countries where they were welcomed, notably Kazakhstan and the United States, with the Texas mining sector emerging as a notable beneficiary during this unprecedented shift in hashrate.

Examining the histograms for 2020 (top left), 2021 (top right), 2022 (bottom left), and 2023 (bottom right) reveals that each year’s estimates of annualized energy consumption exhibit different distributions. Although we observe some discernible patterns, we must exercise caution not to assume this variability recurs every four-year cycle. More data is necessary to establish this assertion. As it stands, our analysis indicates that some years display a bimodal distribution while others display a skewed distribution. The primary takeaway here is that the energy use statistics for each of these four years are distinct, especially for the two years referenced in Chamanara et al.’s analysis.

In the UN study, the authors claimed bitcoin mining surpassed 100 TWh per year in both 2021 and 2022. However, examining the histograms for daily estimated annualized energy consumption shows considerable variation, with many days in 2022 recording estimated energy usage below 100 TWh. We do not deny that the final estimates exceeded 100 TWh in the previously estimated data for these years. Instead, we argue that since bitcoin mining energy consumption isn’t static—fluctuating from day to day or even moment to moment—it’s vital to conduct a deeper analysis to comprehend the origin of this variability and its potential long-term effects on energy usage. Importantly, current updated data suggests annual energy use of 89 TWh for 2021 and 95.53 TWh for 2022.

Lastly, Miller et al. 2022 illustrated that facilities (especially buildings) with significant variations in energy consumption over time are generally unsuitable for emission studies that rely on averaged annual emission factors. Yet, that’s precisely what Chamanara et al. undertook, a common pitfall in flawed models. Bitcoin mining isn’t uniformly operated; it adapts flexibly based on numerous factors, from grid stability and market prices to regulatory environments. It’s time for researchers to start approaching bitcoin mining with this understanding. Had the authors devoted even a modicum of time to studying previously published literature, instead of isolating themselves as noted by Sai and Vranken in their review, they could have acknowledged this limitation in their work.

—

Although I’ve never set foot in a honky tonk before, I found myself squeezed into a cab with fellow attendees at the North American Blockchain Summit. Fort Worth, Texas, perfectly embodied your expectations: large cowboy hats, rugged jeans, and cowboys at every turn. On a chilly Friday evening, Fort Worth felt like a time capsule, busy with pedestrians. The storefronts resembled the quaint shops one might see in The Twilight Zone. I felt completely out of place.

My newfound companions convinced me to try two-stepping. Me, the quintessential California girl, whose physics advisor once quipped that while you can take the girl out of California, you can’t take California out of the girl—two-step? I could hardly distinguish a two-step from an electric slide, and my only brush with country music was a Garth Brooks commercial I caught on TV during my childhood, reflecting his massive popularity in the nineties. That comprises the entirety of my country music experience. Surrounded by quirky gift shops bathed in radiant neon light, I saw a bartender clothed in a diamond-studded belt, brandishing some unknown weapon, a throwback to the guns seen in the 1986 classic, Three Amigos.

Against the backdrop of a country band that wasn’t wholly country, I observed Lee Bratcher from the Texas Blockchain Council demonstrate a grace in billiards that was math-like in precision, routinely sinking balls into pockets during what seemed like his hundredth shot of the night. The familiar clank of billiard balls stirred something inside me. I realized I wasn’t close to concluding my analysis. Somewhere in my hastily taken notes, I recalled that I hadn’t charted the hashrate share over time for the countries analyzed in the UN study. So, at 3:30 a.m., after a swig of club soda that bumped into a photo booth wall, I briefly lost consciousness.

Upon awakening in my hotel room, I was rescued by someone handing me some random fiat currency, ushering me into a cab that took me back to my nonsmoking room in what felt like the heart of the decay that defines contemporary business travel, the Marriott Hotel. Still dazed and bleary-eyed, I allowed the harsh blue light from my screen to bombard my fatigued visage, raising my chances of developing macular degeneration. The analysis continued.

Here are a series of visualizations depicting CBECI mining map data from January 2020 to January 2022. It’s no surprise that Chamanara et al. concentrated on China’s role in energy consumption and its resulting environmental consequences. China’s monthly hashrate peaked at over 70 percent of the network’s total hashrate in 2020. However, by July 2021, this share abruptly dropped to zero before rebounding to roughly 20 percent by the close of 2021. While we still don’t know its current status, industry insiders suggest it likely remains around this figure, indicating that in terms of pure hashrate, the activity continues there despite the prohibition.

Russia also plays a significant role in the discussions surrounding this research. Analysis of the CBECI mining map from January 2020 to January 2022 raises questions about whether Russia was a direct beneficiary of the exiled hashrate. There’s an observable spike, but is that genuine, or are miners merely employing VPNs to conceal their activities? By 2021’s conclusion, Russia’s hashrate had plummeted below 5 percent, falling from a brief high of over 13 EH/s to just above 8 EH/s. When assessing the total annual estimated energy consumption for Russia according to CBECI data, it’s evident that Russia maintained a notable portion of hashrate; however, it’s unclear whether we can accurately infer the current contribution to hashrate and environmental impact based on such limited data.

The most contentious aspect of Chamanara et al.’s analysis pertains to Kazakhstan’s role in energy utilization and its associated environmental footprint. Evidently, the CBECI mining map data reveals a substantial increase in hashrate share, both relatively and in absolute figures. This trend seemingly commenced even before the Chinese ban, with a marked acceleration occurring just prior to and following the implementation. Nonetheless, a sharp decline in hashrate was registered from December 2021 to January 2022; was this a prelude to an impending governmental crackdown in Kazakhstan?

In their research, Chamanara et al. overlooked the contemporary crackdown in Kazakhstan, where authorities instituted an energy tax and licensing system that severely impacted the mining sector, effectively pushing hashrate out of the country. They overstated Kazakhstan’s current significance as a provider of bitcoin’s energy consumption and environmental impact. If they had adhered to the boundaries of their research methods and results, recognizing Kazakhstan’s share during 2020 and 2021’s combined year-end figures may have been plausible. Instead, they not only neglected to account for the 2022 government crackdown, but they also erroneously asserted that Kazakhstan’s hashrate share surged by 34% based on 2023 CBECI data, even though the latter has not been updated since January 2022, and CCAF researchers are still awaiting essential information from mining pools to refresh their mining map.

While I understand I’ve inundated you, dear reader, with a plethora of data, please indulge in another shot of your strongest liquor and let’s examine one final figure. This one illustrates the historical hashrate share of the United States drawn from the older CBECI mining map data. The trend evident in the U.S. closely mirrors that of Canada, Singapore, and what CBECI refers to as “Other countries,” encompassing nations that didn’t make the top ten in terms of hashrate share. There’s a discernible indication that the U.S. absorbed a significant portion of the Chinese hashrate, and this share saw rapid growth during 2021. Recognizing that the CBECI mining map data represents less than half of the network hashrate, I still believe that their share is at least somewhat indicative of the network’s geographical makeup. The distribution of hashrate geographically appears to be significantly influenced by broad macrotrends. While electricity costs are important, governmental stability and favorable regulations play critical roles. Chamanara et al. ought to have conducted this type of analysis to better inform their discussions. Had they done so, they may have realized that the network adjusts in response to external pressures across different times and geographic areas. We require more comprehensive data to make strong policy recommendations regarding bitcoin’s energy use.

—

At this juncture, I found myself questioning whether I was a bitcoin researcher or merely an NPC, trapped in a game where the only scores were the levels of discontent I felt for agreeing to this task. Still, the end of this analysis seemed to be on the horizon, and with enough self-care and therapeutic sessions, I might find clarity about who I was before getting entangled in this circus. Just two days ago, Frank and I had a dispute regarding the appropriateness of Courier New as a font for presenting mathematical equations. I was alone in this rabbit hole now. I gripped the dirt walls confining me and slowly crawled back toward sanity.

Upon emergence from the hole, I snatched my laptop and decided it was time to tackle the study’s environmental footprint methodology, complete this analysis, and tie everything up neatly. Chamanara et al. claimed their methodology aligned with that used by Ristic et al. (2019) and Obringer et al. (2020). This approach is flawed for several reasons. First, the footprint factors are usually employed to assess the environmental impacts of energy generation. Ristic et al. developed a metric called the Relative Aggregated Factor that incorporated such factors, allowing for the evaluation of electricity generators like nuclear or offshore wind. The purpose was to be aware that while carbon emissions from fossil fuels are the primary driver for transitioning energy goals, we must also avoid replacing fossil fuel generation with alternatives that might introduce different environmental issues.

Second, Obringer et al. took many factors listed in Ristic et al. and combined them with network transmission factors from Aslan et al. (2018). This was questionable since Koomey was a co-author, and it’s not surprising that in 2021, Koomey co-authored a commentary with Masanet, addressing Obringer et al.’s methods. Koomey and Masanet criticized the assumption that short-term demand shifts would lead to immediate and proportional changes in energy consumption. This critique is applicable to Chamanara et al., examining a period when bitcoin was approaching an all-time price high in a unique economic context (low interest rates, COVID stimulus checks, lockdowns). Koomey and Masanet made clear that neglecting the disproportionate relationship between energy and data flow in network equipment can lead to inflated environmental impact assessments.

Above all, we have yet to fully characterize what this correlation entails for bitcoin mining. Demand for traditional data centers is dictated by required computing instances. What represents an equivalent for bitcoin mining, given that block sizes remain constant and the block timing is adjusted biweekly to maintain an average of 10 minutes between blocks? This deserves more attention.

In any event, Chamanara et al. appeared unaware of the critiques aimed at Obringer et al.’s approach. This oversight is problematic, as Koomey and Masanet have laid the foundations for energy research related to data centers. They ought to have known not to apply these methods to bitcoin mining, as differences between the industry and conventional data centers still exist. There’s a wealth of knowledge in data center literature that bitcoin mining researchers should leverage. It is both disappointing and exhausting to witness publications that disregard that reality.

Ultimately, this cycle of flawed research must come to an end. Brandolini’s Law is real, and the imbalance of misinformation is genuine. I earnestly hope this new halving cycle will be the last time I address poor-quality research. While I was writing this report, Alex de Vries published yet another misleading paper on bitcoin mining’s “water footprint.” I haven’t delved into it yet, nor am I certain whether I will. But if I do, I promise I won’t pen an extensive 10,000-word critique. I’ve made my arguments and fulfilled my obligations regarding this genre of academic discourse. It’s been an entertaining journey; however, I believe it’s time to prioritize my well-being, indulge in some evenings of wholesome binge-watching, and dream of the unfathomable.

—

If you found this article thought-provoking, please visit btcpolicy.org to access the full 10,000-word technical assessment of the Chamanara et al. (2023) study.

This is a guest contribution by Margot Paez. The views expressed belong solely to the author and do not necessarily reflect those of BTC Inc or Bitcoin Magazine.