How to Burn Money in the Cloud // Avoid AWS, GCP, Azure Cost Disasters

when you decide to move to the cloud oneof the first things you have to dois give them your credit card that'sbecause they're giving you access to amassive network of computers worthbillions upon billions of dollars whenyou write inefficient code on your owncomputer it's usually not a big deal thecpu fan might spin a little faster orthings might run slower when you run outof rambut in the cloud it's a whole differentball game that recursive function youwrote can now scale infinitely aroundthe globeblowing up cpu cores and melting solidstate drives in the processand someone is going to have to pay forthat mess in today's video we'll look ata real life story about how a startupjust days away from launch racked up a72 thousand dollar google cloud billstand it's gone didn't do too well it'sgone poofwe'll find out how that's even possiblebut more importantly we'll look at waysto prevent something like that fromhappening to youif you're new here like and subscribethen let's get right into itimagine spending months pouring bloodsweat and tears into your startupyou're just days away from the biglaunch doing some final testing here andthereyour product is ready to go you justneed to scrape some data from the web toenrich the user experienceyou throw together some ad-hoc code toget the job done dockerize itthen deploy it to a serverless platformlike google cloud runit only needs to run for a few hours soyou deploy it then call it a nightyou wake up to a couple of interestingemails our firebase project was upgradedfrom the free planto the pay-as-you-go plan we must haveenabled an api that requires billingwhich automatically upgraded the plannot to worry we already have a budgetalert in place oh nothe next email tells us our seven dollarbudget has been exceededthe third email tells us our credit cardwas declined i wonder why that would belet's open the billing panel in googlecloud no god pleaseno no a bill of over five thousanddollars in just a few hoursit has to be a mistake let's refresh thepage and prayno no five minutes later the bill's atfifteen thousand dollarsafter twenty minutes twenty fivethousand dollars the final bill settledup after about two hours just short ofseventy two thousandincreasing your startups burn rate byabout six hundred dollars per minuteno if you're a bootstrap startup youmight as well make it ten milliondollarsto make matters even worse this companyhad multiple productswith multiple google cloud projects tomanage them because things got out ofcontrol so quicklythey had to disable all of theirprojects and that's the story of how onelittle recursive function took down anentire businessi declarebankruptcy luckily the story doesn't endtherethe company immediately contacted googlecloud to make them aware of the issueand let them know that they did notintend to purchase this much computingpowerafter 10 days of deliberation google didthe not evil thingand credited the entire 72 thousanddollar bill that must have been a hugerelief for the company and was a big winfor independent developers around theworldbut keep in mind there's no guaranteeyou'll get the same kind of credit whenyou rack up a massive bill the beststrategyis to not rack up a massive bill in thefirst place however that's easier saidthan donei've actually racked up two massivebills myself one timei accidentally exposed my aws api key ina public git repoa few months later i had a 5 000 awsbill because a hacker scraped the keyand then used it to spin up a bunch ofcompute instancesluckily they gave me a credit for mybill as well it is very reassuring thatthese big cloud companies would ratherkeep you as a long-term customerthan try to bankrupt you over an honestmistake but a bill of 72 000is crazy i've never seen a bill get thathigh that quickly and we should all bethanking announced today for sharingthat story with uscheck out their web app for a greatexample of angular and firebase in thewildthen let's dive into the technicalreasons for the massive bill and moreimportantlyways to avoid it yourself tip number oneset up a budget alerteven if you're on the free plan ofsomething or in the free tier once youdeploy code to the cloud there aredifferent ways you might exceed the freetier without knowing itit's not common but it's still always agood practice to have a budget alert seton your account globallythe company did this right by setting abudget for seven dollarsthe problem in this case though was thatthe cost increased so quicklyan alert will only send you an email itdoesn't actually shut down anythingwhen the budget has been exceeded it'sactually kind of crazy that there's notan easy way to shut down serviceswhen the budget has been exceeded that'sbeen a major criticism of the cloud foras long as i can rememberhowever there may be hope in october2020 aws released a new feature calledbudget actions where you can define anaction based on a budget threshold likeshutting down a server once you've spenttoo much moneyin addition on google cloud you canlisten to budget events with pub suband that means you can write a littlebit of code to create a kill switchthat disables billing if the budget isexceeded now let's move on to the nexttipdo not create infinite loops in thecloud do not createinfinite loops in the cloud do notcreate infinite loops in the cloudwhen it comes to serverless computingeverything is event-basedsome kind of event happens in the cloudlike an upload to a storage bucketwhich triggers a cloud function foradditional back-end work it's reallyeasy to shoot yourself in the foot herethoughif your function leads to the same eventthat triggered it in the first placefor example a cloud function istriggered by a storage uploadthen you upload a file to the storagebucket again and the functionthus triggering another function andagain and again and forever until theend of timeor until your bank account is emptythat's what we call an infinite loop andit's probablythe number one cause for runaway costunexpectedlyespecially if your infrastructure isconfigured to handle huge workloadsthat brings me to tip number three don'tconfigure things to scale when you'rejust messing around our startup used aservice called cloud runwhich are serverless functions thatsupport docker containersby default cloudrun will scale to a maxof 1000 instancesnow if we take a look at thearchitecture for the web scraper you'llnotice it starts with a manual requestto the cloud run endpointeach run scrapes url and saves theresults in firestorethen it takes all the links it found ina scraped webpage and triggers anothercloud run to process themif we look at this as pseudocode what wehave here is a recursive functionbut a recursive function with nostopping point which leads to stackoverflowor in this case cloud overflow which isa brand new word that i just coinedbecause the infrastructure was set up toscale it resulted in a 116 billion readsto the database and 33 million writesalong with 16 000 hours of cloud runcompute timethe moral of the story is to understandhow different services scaleand configure them to not scale if youdon't need that functionalityin the case of cloud run if the maxinstance option was set to 1instead of 1 000 their bill would havebeen 72 dollarsinstead of 72 000 but in production youlikely do want things to scale andsometimes you won't know how bad anissue isuntil your app does scale which bringsme to tip number fourconsider how algorithm complexityaffects your cloud computing usage awhile back i made a videoabout how to not get a 30 000 firebasebill which was another case study abouta company that went viraland a flawed data model caused theirbill to accelerate exponentiallyallow me to explain they built acrowdfunding app and in the appthey calculate the total amount donatedto a crowdfunding causethey perform the calculation client sideby reading all the donations from thedatabase for that causenow let's imagine you have five peoplewho donated to a cause and a hundredvisitors to that pagethat would equal 500 document reads nowlet's imagine you have a hundredthousand visitorsthat'd be about 500 000 reads whichstill isn't too badbut now let's imagine you have 5 000donations to a causemultiply that by 100k users and you nowhave 500 million document reads or inother wordsat the current traffic level each newdonation adds an additional hundredthousand document reads to the totalworkloadit just doesn't scale well a betteroption would have been to aggregate thedata on the backend with a cloudfunctionrather than calculate the total clientside it could be done on the back endso that each visitor only has to readone document instead of 5 000 documentsnow i realize stories like this arepretty alarming but firestore isactually very inexpensiveand because it's so easy to use you maynot realize you're making a bad datamodeling choice until it's too lateusing the right data model can go a longway and you can learn more about that inmy data modeling coursebut yet another way you might experiencerunaway cost is by failing to secureyour appand that leads me to tip number fivemake sure your app is securemost importantly make sure to protectany secret api keyslike i mentioned earlier exposing my awsapi key on githuballowed a hacker to take full controlover my account but there are many otherways you might fail to secure yourinfrastructurewhen it comes to firestore you need todefine security rules to define who hasaccess to what in the database and ofcourse i have a full course on that aswellleaving the database open means anybodycould read or write to itnot only does that leave you vulnerableto a data breach but also to hackersthat just want to drive up your coststhe same principle goes for anyapplication that is available on the webwhen you have a service that's availableon the internet it means someone canfind that urland exploit it i'm going to go ahead andwrap things up there please make sure tolike and subscribe and if you'rebuilding an app with firebase or googlecloudconsider becoming a pro member atfireship io thanks for watching and iwill see you in the next one