Who is man enough to explain "The Cloud" to me?

Discussion in 'Game Development (Technical)' started by Raptisoft, Jul 7, 2011.

  1. Raptisoft

    Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    804
    Likes Received:
    0
    Hi all,

    I've been googling this for a couple days, on and off, and I always get the information I don't need, and a whole lot of online sales talk.

    I want to write myself a little program that will synchronize a folder on my computer with an Amazon Cloud account. Basically, I'm trying to write a miniature version of Dropbox for myself, in the hopes that I can eventually offer similar services to my users so they can back up their data and transfer it and all that user stuff that makes writing a two month game turn into a three year ordeal these days, and if you don't have it someone gives you iTunes ratings like "awesome game, I've been playing it for 35 days straight, but you can't back your data up into the cloud, and no multiplayer-- one star."

    So I assume you get a cloud account... and then you... write data to it? How do you do it? Like, do you say "Here's a chunk of data, record ID 'wassamattau?'" Or do you do database calls? How exactly does this stuff work, in its simplest form?

    Can I stash a simple array of data in the cloud? Like, could I put a directory structure into the cloud simply by saying here's a chunk of data, store it with id C:\MYFOLDER\GIRLS\CHICKIE.JPG, like, dumping it there with LibCurl or something?

    It's super simple, right? You don't need to use any weird system calls, it's all post to a URL or something, right? Wrong? In some cases?

    A moron-level tutorial would be most appreciated. If you're a young hotshot programmer, pretend you're explaining it to your mother, since I'm 39 and the world got all new and frightening in the last six years or so.
     
  2. papillon

    Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    2,193
    Likes Received:
    0
    Man enough? Chickie? Mother? Hrmph! :)

    ... I don't really get clouds either, though. I must be old.
     
  3. mooflu

    mooflu New Member

    Joined:
    Oct 23, 2007
    Messages:
    21
    Likes Received:
    0
    Wikipedia refers to a good definition:
    "Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction."

    With the cloud you generally can't point to a physical box and say, the data is stored there or the service is running there etc. It's just somewhere in that fluffy thing.

    Users shouldn't care, although they sometimes think they do. They usually just want to be able to run their game on device A and then later on continue on device B (i.e. user profile, scores, saved games, etc. stored online).

    As a developer you care because the cloud scales, otherwise you could just use a shared hosting setup.

    API prolly provider specific. For example, Google Apps Engine has different ways to store data. For large binary chunks it has a Blobstore and (experimental) file API (See http://code.google.com/appengine/docs/python/blobstore/overview.html#Writing_Files_to_the_Blobstore). That's server side, it's still up to you to define how the data gets from client to server (the Blobstore page also has a http POST example that uploads a file and stores it as a blob).

    HTH
     
  4. puggy

    Original Member

    Joined:
    Mar 10, 2005
    Messages:
    195
    Likes Received:
    0
    http://en.wikipedia.org/wiki/Cloud_computing has good basic info.

    Basically it's just having access to the same thing over multiple servers. A good example is google search engine, type in google.com and your directed to one of thousands of servers, all with the same data on them. one server breaks down, no matter you got plenty more so it won't affect your site to the users.

    Bringing it to the indie game scene, imagine you launch a game, you have your website on a dedicated server, first 2 weeks your getting 10k hits per day on your site, everything seems fine, then a game review site features your game and all of a sudden you have 10k hits per 10 minutes, your server can't handle the load and is shut down, no one can buy your game now and you have wasted all that publicity.

    If it was on a cloud service, they may have at first 3 servers with your website on it, there isn't a big strain as the load is being diverted to 3 servers, however if it starts overload them, there will be procedure in place to copy your website to a few other servers with little load, meaning no loss of service at all.

    Of course for this security you'll pay through the nose but then that's why a lot of hosting companies are hyping this up because they make more money from this than normal bog standard hosting services.
     
  5. jpoag

    jpoag New Member

    Joined:
    Mar 15, 2008
    Messages:
    806
    Likes Received:
    0
  6. Jack Norton

    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    5,130
    Likes Received:
    0
    Google Apps is probably the easiest solution. For other services I am not sure if they scale automatically? For liquidweb for example you pay a fixed monthly amount and I haven't seen anywhere a text saying "if your game is slashdotted and you get 50k uniques/hours we will automatically use 10 servers".
     
  7. Gary Preston

    Original Member

    Joined:
    Aug 5, 2005
    Messages:
    239
    Likes Received:
    0
    Google App Engine gets my vote too. It's simple and free to get started and easy to adjust billing options when you really need to be able to scale up to higher traffic.

    There's quite a few docs on the google site in particular check the What is Google App Engine page and the Tutorial

    Sign up for a google apps account and an app engine account (requires a mobile phone for activation). Then you can create a test app and install the SDK for Python, Java or Go. There are quite a few examples online too, although imo the easiest way to get started with it is to just follow along with the Python tutorial and make a simple page then upload that to your test app and play around with the other api's like the datastore.

    That'll cover setting up the app config, how to route requests to your script handlers and a rough intro to the api and Django templates.

    What makes this more "Cloud" than just normal hosting, is the replication and instance management that google automatically does. That's why there's some restrictions on what you can/can't do and how you have to do it.
     
    #7 Gary Preston, Jul 8, 2011
    Last edited: Jul 8, 2011
  8. Indiepath

    Indiepath New Member

    Joined:
    Aug 22, 2004
    Messages:
    999
    Likes Received:
    0
  9. Raptisoft

    Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    804
    Likes Received:
    0
    Just to clarify... I understand what cloud computing is... what I want to know is, specifically, how to do this:

    void WriteDataToCloud(char *theID, char *theData, int theDataLen)
    {
    Assuming "theID" is the record ID I can use to fetch/store this data (is that how it works?)...
    ...what do I put here?
    }
     
  10. Gary Preston

    Original Member

    Joined:
    Aug 5, 2005
    Messages:
    239
    Likes Received:
    0
    You send files the same way you would if you were uploading a file to a web server. Encode it and place it into a post request. On the server side you'd have a rule in the app.yaml that detects the given upload url and hands off to a script which can get at the file data and use the DB api to store it.

    If you're not using a web browser as the "client" but instead wanting to do a post from within your game, then you'll need to form the html post request yourself (or use a library that wraps it) instead of just using a html form to handle it.
     
  11. Raptisoft

    Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    804
    Likes Received:
    0
    Thanks Gary, now we're getting somewhere...

    So let me see if I understand this:

    Right now, I have a LibCurl wrapper that lets me dump a file to my web server. Php on the server end grabs the temporary file that the post makes, and puts it into a database in my server.
    Are you saying that the php itself on my server is going to have to dump it into the amazon cloud? I.E., I don't upload to the Amazon Cloud, I upload to an intermediate place, which then puts it in the cloud?

    And if that IS how it goes, does anyone have an idea where I can find an example of the php end of things?
     
  12. Gary Preston

    Original Member

    Joined:
    Aug 5, 2005
    Messages:
    239
    Likes Received:
    0
    Not quite.

    If you replace "my server" with App Engine "Server" and Php/Mysql with Python/Java or Go running on the app engine and using the Datastore or Blobstore api, then yes.

    When you use app engine, you no longer need your server and in turn php/mysql.

    For example, you'd setup a cname for myapp.mydomain.com to direct to ghs.google.com. Any time you connect to that URL (via web browser or another client that forms http requests which I believe includes an app using libcurl) , you'll be directed to a google app engine server, which server will depend on server loads, geographic location etc

    When a request comes in app engine will look at a app.yaml file you've previously uploaded to find out which script to run based on the URL routing you setup in that config file. The "script" can be written in python, java or go. You upload the yaml and all the scripts in one go and app engine will ensure they're distributed amongst however many servers/data centres they deem appropriate.

    Your script is then run and given access to the http url and other parameters like the get/post request variables. Your script extracts whatever data it needs (much like you would with a PHP script) but instead of then communicating with MySQL you use the app engine Datastore or Blobstore API to push data into a DB. The changes made to the DB are transparently replicated by app engine to all the other servers.

    Your script can then produce a response to send back to the calling client, which may be a complete html webpage built using templates or built manually.

    Whilst you could upload to your server as an intermediate and then have PHP hand off to app engine, it would negate the main reason for using app engine, which is scalability when a traffic spike occurs. Your server would be a bottleneck.

    You'd usually (there may be exceptions I guess) communicate directly with the app engine "server". You might then queue things up in order to pass onto another 3rd party server that you need to interact with, but in this case app engine would be the intermediate.

    I'd suggest signing up for an account and trying to get a simple website page been served. Then you can play around with the script to handle post requests, save data to the datastore, retrieve/display the data etc After that it should be pretty clear how it works. Getting data to/from your app is then just a case of issuing the html get/post requests from your app.

    Hope that clears things up a little, if not, search for "app engine" on youtube and watch a few of the videos or a more detailed but old video from 2009.
     
    #12 Gary Preston, Jul 8, 2011
    Last edited: Jul 8, 2011
  13. Raptisoft

    Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    804
    Likes Received:
    0
    Thank you Gary, very helpful. You've given me enough to start the right searches to get this going on my own, I think!
     
  14. jpoag

    jpoag New Member

    Joined:
    Mar 15, 2008
    Messages:
    806
    Likes Received:
    0
  15. bantamcitygames

    Administrator Original Member Indie Author Greenlit

    Joined:
    Jul 27, 2004
    Messages:
    1,732
    Likes Received:
    78
    I'm not putting my stuff on dropbox after this:
    http://blog.dropbox.com/?p=821

    I've not used Amazon or App Engine, but at work we've use the Azure cloud (Microsoft) and you would make a call to open a BlobStorage connection with your credentials and then you would make a call which basically uploads the file and then close the BlobStorage connection. I'm assuming the other ones are fairly similar, but if they support a simple POST then that would work too.
     
  16. Jack Norton

    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    5,130
    Likes Received:
    0
    Wow. Luckily I upload the sources already encrypted so they can't see them in any case. Seriously you can't upload files "in the clear" on internet! I first encrypt LOCALLY and then upload :D
     

Share This Page

  • About Indie Gamer

    When the original Dexterity Forums closed in 2004, Indie Gamer was born and a diverse community has grown out of a passion for creating great games. Here you will find over 10 years of in-depth discussion on game design, the business of game development, and marketing/sales. Indie Gamer also provides a friendly place to meet up with other Developers, Artists, Composers and Writers.
  • Buy us a beer!

    Indie Gamer is delicately held together by a single poor bastard who thankfully gets help from various community volunteers. If you frequent this site or have found value in something you've learned here, help keep the site running by donating a few dollars (for beer of course)!

    Sure, I'll Buy You a Beer