RedMallorca LAB|Blog LAB para pruebas

Introduction

I'm a big fan of pet projects. You know the ones I mean: the projects we love to start and hate to finish. The two-week remodeling gig that takes two years. The '69 Mustang sitting on cinderblocks in the back yard while seasons rotate. The unfinished novel lurking on the nether regions of your hard drive. And for programmers and poker players around the world, a million unfinished tools and libraries ranging from the ingenious to the depressingly obscure.

Today, I’d like to talk to you about a pet project which is actually worth your time.

When I say worth your time, I mean a project which is financially, professionally, and personally rewarding in a way that pet software projects almost never are. Everybody likes money, of course; and this project, properly executed, is worth a small fortune. In fact, building and successfully running even a flawed and break-even version of the bot for several years is worth a decade of paychecks (for most people). But the money is (or should be) an afterthought, a nice-to-have.

The real reason I can suggest this project is as follows:

Building an online poker bot will give you a deep knowledge of two lucrative domains, programming and poker, and enough day-to-day working familiarity with those domains to position you, given some effort on your part, for the work of a professional software developer and/or poker player.

How does building a poker bot make you a better poker player? Because it forces you to confront, understand, and codify advanced poker strategy in a way that few of your opponents will have done. How does building a poker bot make you a better programmer? Because a bot involves a balanced coordination of techniques that span the breadth and depth of software development, from operating system nuts and bolts to abstract A.I., from user interfaces to multi-threading.

So to a certain degree, I’m the little devil on your left shoulder telling you to take the path less travelled by.

If I get my way, not only will you be building an online poker bot, you'll be playing poker seriously, for meaningful stakes - if you're not doing so already. You'll be programming seriously - if you're not doing so already. You'll leverage your programming skills to make you a better poker player. You'll leverage your poker-playing skills to make you a better programmer. And you'll leverage your natural (and healthy) desire for money to keep you interested in the process even when the going gets rough - and it will.

Looking at the source code for the bot, I'm reminded of another benefit: the creation of a generic code base which can be used to solve a great many poker problems other than online poker botting. If you structure this code base precisely, you don't have to put all your eggs in one basket. You can cautiously develop in the direction of an online poker bot, while giving yourself the option of branching off into other poker-related projects, many of which have significant commercial appeal (as the author of Poker Tracker could tell you).

Now, the last thing this series is, is a get-rich-quick scheme. But let's pull that copy of Theory of Poker off the shelf, blow the dust off the cover, and turn to the chapter on Implied Odds.

Implied odds are based on the possibility of winning money in later betting rounds over and above what is in the pot already. More precisely, your implied odds are the ratio of your total expected win when your cards hit to the present cost of calling a bet.

Implied odds dominate big-bet forms of poker such as No Limit and Pot Limit, and even affect play in Fixed Limit games, to a degree. But implied odds also apply to real-world scenarios of risk and reward. For example: let's say you're considering whether or not it's worth your time and effort to start building an online poker bot.

You can't simply calculate the likelihood of your being able to produce a winning poker bot, factor in the number of hours played, and so forth. You have to look at the hard-to-quantify value of the knowledge itself, and its effect on your career and life. You have to factor in the value of serendipity. In other words, you have to consider what you might call the implied odds of the situation.

• Will the bot make money? By winning money from other players? Via promotions? As a product?
• Will the skills I learn qualify me for a job I never could’ve gotten otherwise?
• Will I get addicted to poker and end up playing for a living, in Vegas, California, London?
• Will I meet a person or persons who will introduce me to other opportunities?
• Will I end up taking the bot’s source code and selling it, or developing a full-fledged commercial application?
• Will I enter my bot in one of these “botting competitions” I’ve heard about?

Taking those factors into account, I think you’ll come to the same conclusion I did: there’s considerably more at stake here than making a few bucks running your bot on Poker Stars. The implied odds of building an online poker bot are huge, even if your bot never turns a profit.

If that hadn’t been the case, I never would’ve sat down to write that first line of code…

Underground Poker Meets Best-Practices Software Development

This whole thing started, appropriately enough, in Dallas, Texas, where a booming IT corridor intersects a thriving underground poker scene - or what used to be a thriving underground poker scene.

All-night poker clubs in back-alley office lots and basements. Places with names like 6th Street, MMO's, Executive Game, DTD, DPO, Third Pair. Full-service underground casinos with professional dealers. ATMs. Widescreen TVs. Dollar-a-minute table massages. $60,000 bad beat jackpots. On-site kitchens stocked with every fried food known to man. And the #1 draw of most Dallas poker games: free beer.

Times have changed in Dallas and many of these clubs have closed their doors. For a few reasons:

• The uncapped 10% rake structure of many low-limit Dallas games, combined with a way-above-average dealer toke, plus the ubiquitous bad beat jackpot, sucking an additional $1 from every pot.
• The opening of multiple major (legal) cardrooms an hour north, across the Oklahoma state line.
• The recent return of robberies at gunpoint, unheard of in Dallas since the days of the road gamblers.
• Increased scrutiny of the Dallas underground poker scene in the local news.
• The reversal of local law enforcement’s long-standing policy of turning a blind eye towards local poker games.

But once upon a time, you could play no-limit Hold’em from dusk ’til dawn, seven days a week, at fifty different clubs around town, for buy-ins ranging from $40 to $5000 or more. It reminds me of a movie every poker player either loves, or loves to hate: 

Playing poker in Dallas a few years ago, you felt like you were living a scene from Rounders, except that the situation in Dallas went way beyond anything Rounders ever portrayed. The sudden mass influx of new money into the game - thanks in part to Rounders, thanks in part to online poker, thanks in part to Moneymaker, Raymer, Varkonyi - meant that anyone with a spare $30,000 lying around could start up a club and recoup their investment in a matter of a few months. People were literally renting out warehouses and throwing weekly freeroll tournaments to lure new players into the cash games (which of course, are where the house makes all its money). These freerolls drew hundreds of players - not a big deal in Vegas, perhaps, but this is a state where running a raked poker game was and is a class A misdemeanor.

 § 47.04.  KEEPING A GAMBLING PLACE.  (a)  A person
commits an offense if he knowingly uses or permits another to use as
a gambling place any real estate, building, room, tent, vehicle,
boat, or other property whatsoever owned by him or under his
control, or rents or lets any such property with a view or
expectation that it be so used.
    (b)  It is an affirmative defense to prosecution under this
section that:   
    (1)  the gambling occurred in a private place;                                
    (2)  no person received any economic benefit other than
personal winnings;  and
    (3)  except for the advantage of skill or luck, the
risks of losing and the chances of winning were the same for all
participants.
    (c)  An offense under this section is a Class A misdemeanor.

That’s just shy of a felony, and trust me when I say: the people running these games weren’t exactly hardened criminals. They were guys with landscaping businesses, law practices, and programming jobs.

Anyway, there was so much new money (also known as dead money) on the tables, you didn't have to play particularly well, in order to win. All you had to do was be patient. The strategy went something like this:

• Wait for a monster hand.
• Get all your money in against players who don’t notice or don’t care that you haven’t played a hand in four hours.
• Rinse and repeat.

It's the (exploitable) style of play known in poker circles as weak-tight and that, folks, is all it took to grind out a decent living in those underground Dallas poker games of yesteryear. This style, absent tilt, is still good enough to win in today’s low-limit online games, though as we’ll see, the tight-aggressive and loose-aggressive styles are really where it’s at.

I mention all this because very early on it occurred to me: there's got to be a way to extract this ABC style of poker into a piece of software and then hook that software up to an online poker site. And if you’re a programmer who plays online poker seriously, don’t pretend you haven’t thought the same thing.

After all, why else do we learn to program, if not to be able to solve real-world problems?

An Introduction to Advanced Poker A.I

Many of you have commented that the Input (getting information from the poker client) and Output (clicking buttons and otherwise manipulating the poker client) stages are low-hanging fruit: tedious, but not rocket science. The Processing stage, on the other hand - the cognitive element of the bot - is another story.

Morpheus: This is your last chance. After this, there is no turning back. You take the blue pill - the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill - you stay in Wonderland and I show you how deep the rabbit-hole goes.

Artificial intelligence is a huge and fascinating subject, even when applied to the (relatively) narrow domain of poker. A sample of recent papers from the University of Alberta’s Computer Poker Research Group (CPRG) should give you a taste:

• Computer Poker
• Dealing with Imperfect Information in Poker
• Probabilities and Simulations in Poker
• Poker as a Testbed for Machine Intelligent Research
• Opponent Modeling in Poker (PDF)
• Using Selective Sampling Simulations in Poker
• Learning To Play Strong Poker
• Using Probabilistic Knowledge and Simulation To Play Poker
• Improved Opponent Modeling in Poker
• Opponent Modeling in Poker: Learning and Acting in a Hostile Environment
• The Challenge of Poker
• Approximating Game-Theoretic Optimal Strategies for Full-scale Poker
• Opponent Modeling and Search in Poker
• A Tool for the Direct Assessment of Poker Decisions
• Optimal Unbiased Estimators for Evaluating Agent Performance
• Algorithms and Assessment in Computer Poker
• Postgame Analysis of Poker Decisions
• Robust Strategies and Counter-Strategies: Building a Champion Level Computer Poker Player
• Using State Estimation for Dynamic Agent Modeling

Unfortunately, most cutting-edge poker A.I. is beyond the reach (given the average person's schedule) of the average programmer/poker-player:

I have lot of respect for the scholarly work that the CPRG and others have done, and I think it's the kind of work which will end up producing the very strongest poker intelligence. But I'd recommend you stay away from the advanced, graduate-level A.I. (unless you happen to be an enthusiast or expert in the field) in favor of learning basic and advanced poker strategy cold. Ultimately that will get you more bang for your mental buck.

Botting Rule #16: None of the techniques used by the CPRG or other organizations are necessary to produce a winning online poker bot.

Now, those are some fightin’ words, so let me qualify them. The CPRG approach has always been a game-theoretic approach: finding optimal and/or exploitive strategies which attempt to model the game of poker “from scratch”. That means a lot of abstract math and statistical modeling, a lot of terms like Nash equilibria and Best Response, and it means relatively little domain-specific knowledge of poker as it's played for real money in today's online cardrooms:

• How to incorporate (for example) Poker Tracker statistics into the bot’s decision-making.
• How to take advantage of player loyalty incentives and promotions.
• How straightforward, ABC poker play wins in low-limit online games.
• Convenient rules of thumb (if you flop a set, you’re almost never making a mistake to simply get all your money in on the flop) and how they can simplify processing.
• Table selection, table selection, table selection (the poker equivalent of location, location, location).

That's not to say that the people at the CPRG don't "know" poker, or that they couldn't build an online poker bot which would beat the pants off of any other online poker bot that had the temerity to compete against them. It's just that they have a different goal in mind: creating a generic, from-scratch poker intelligence which is capable of competing at a world-class level using pure A.I. That’s miles and miles beyond what it takes to produce a profitable online poker bot. And while we're not going to rule out any work that the CPRG (or other organizations) have done, at the same time, let's be realistic. How much time can you devote to building an "optimal" online poker bot? Two years? Ten? Can you devote six solid months to learning (or re-learning) the math? No?

Then start small, by choosing one or more of the following shortcuts:

• Don’t require that your bot actually win. Instead, play the online promotion game so that it makes money even if it’s a slight overall loser.
• Force the bot to play short-stacked, and to leave the table when it’s no longer short-stacked.
• Specialize the bot for a very specific game, such as "low-limit, heads up, No Limit Hold'em tourneys".
• Force the bot to play at a full table (or short-handed), and to leave the table when that changes.
• Allow the bot to rely on occasional human assistance, or even to notify (through a loud buzzer) a nearby human when it needs assistance.
• Use toy games (no-fold, half-street, static card, etc) to simplify analysis.

Getting the bot up and running on “training wheels” in this way is one of the things that makes this project feasible for the tinkerer or botting enthusiast.

Iterative, Incremental Development

We’re not going to rely on the above shortcuts exclusively. We want to have our cake and eat it too: meaning, we want to produce a bot which is capable of taking these shortcuts, but which has enough cognitive flexibility to leverage more sophisticated approaches, as we code them, or as they become available.

Botting Rule #44: In early versions of the bot, rely on inventive shortcuts to cut into the complexity of the game, and schedule these shortcuts as milestones on the path to the development of a full-fledged bot.

It's not much fun to spend two years coding away in isolation only to discover upon completion that your bot is ridiculously weak, and overly complex given how poorly it plays. Delineate the work into bite-sized chunks which you can deploy and test immediately. 

You working programmers out there already know the score on this one. Formal software development methodologies from RUP to Agile have one thing in common: iterative, incremental development. Even if you've never worked with a formal development methodology, you know that every month or so, customers, clients, managers, coworkers - somebody of importance - wants to see what you've been up to. When it comes to building a poker bot, likely as not that person will be you. And that means it's more important, not less important, to set bite-sized, verifiable development goals. Otherwise it's easy to get lost in the labyrinth of Windows application development, poker domain modeling, and generic A.I. which comprise a bot. In fact, it's impossible not to get lost if we don’t break the problem down into manageable pieces.

And one of the big tools we'll use to allow that (when it comes to the bot's A.I.) is an extremely simple form of what's called a multi-agent system, or MAS.

Pluggable Intelligence and the Multi-Agent System

This is a subject (A.I.) that we'll be returning to again and again; but for now, I want to lay out some basic architecture. This isn't the only approach, and I make zero representations as to whether it's the best, or even a particularly good, approach. But it is a working approach which has undergone the trial-by-fire of competitive, real-money play.

What we’re looking for is a generic mechanism which allows any poker processing logic or intelligence whatsoever - from the simplest of “hot and cold” probability analyzers to the cutting-edge artificial brain of tomorrow - whether written by us or by a third party - to participate in the decision-making process for a given game situation.

Botting Rule #4: Pluggable intelligence is a key requirement for an evolving bot.

In order to do that, we have to design the bot's cognitive component such that it's capable of incorporating and sublimating any arbitrary poker A.I. In effect, we want to build a simple multi-agent system:

A multi-agent system (MAS) is a system composed of multiple interacting intelligent agents. Multi-agent systems can be used to solve problems which are difficult or impossible for an individual agent or monolithic system to solve. Examples of problems which are appropriate to multi-agent systems research include online trading, disaster response, and modeling social structures.

Speaking in more precise terms: consider creating a programmatic interface (I called mine DecisionModule) which allows any arbitrary component or application to participate in the voting process ultimately responsible for producing a discrete output…

• fold
• check
• bet
• raise
• call

…for a given game situation. Each "Decision Module" can be thought of as a distinct intelligent agent, operating independently of or in cooperation with all other such agents which comprise the bot. These agents might be source-level classes that you write yourself, or they might be third-party applications or DLLs for which you've written an adapter.

For example, let's say a very strong commercial poker playing application is released, and it has a programmatic interface. With a pluggable MAS, you can leverage that application's intelligence as a pluggable component inside your bot, running in parallel with whatever other components you've created or acquired. 

In practical terms, that means you can have a simple "hot and cold" probability analyzer running next to tomorrow's cutting-edge third-party "artificial brain"; from the bot's perspective, each such component is a black box. Table state is passed into a sequence of Decision Modules, each of which conforms to a uniform interface. Together, they produce the ultimate decision to call or fold, in the form of a distribution which specifies (for example):

• Raise 80% of the time
• Call 20% of the time

The specific modules shown above are just by way of example, and of course, you can design this out any way you please. The important thing is to promote the concept of the multi-agent system to a first-order architectural abstraction within the bot. This approach has a number of advantages, for the A.I. hobbyist in particular:

• Support for any existing or future poker A.I., winning system, or method.
• We avoid hard-coding our bot to a particular solution, which may prove to be naive.
• We’re able to drop in new plugins as we develop them or find them on the open market.
• Any single plugin can contain any level of sophistication, from simple toy game analysis to the cutting-edge heuristics of tomorrow.
• We have the ability to assign weights and other valuations to components, add and remove components, and otherwise tweak the voting process.
• We decouple the cognitive elements of the bot from the sensory and motive elements.
• Analysis can be run either synchronously, in the foreground, or asynchronously, in the background (worker thread semantics).

Once that's in place, we can start leveraging specific algorithms in the pursuit of a plus-EV bot, starting with simple rules-based or simulation approaches and eventually graduating to some more sophisticated abstractions. Again, an iterative, incremental development path studded with bite-sized, achievable goals is the way to the money.

Modeling the Poker Domain

In the first part of this series, we decomposed the bot into the three classic stages of information processing: Input, Processing, and Output. I noted that each stage of processing should run independently of the other stages.

• The Input stage should know nothing about poker; it just tries to extract arbitrary data from a given application using a set of rules.
• The Processing stage should know nothing about screen-scraping, API hooking, DLL injection, or input simulation. It's job is to take an abstract model of a poker game and analyze it.
• The Output stage should know nothing except that it’s been instructed to simulate certain mouse clicks at specific times.

That's a good architectural division which it's worth preserving. But in practice we'll find that there are other even more meaningful divisions. In particular, the division between "generic poker things" and "online poker things". 

Again and again, you’ll find that online poker things want to derive from generic poker things. In the diagram above, we can see that StarsPokerTable inherits from OnlinePokerTable, which inherits from PokerTable. We can use true Liskov substitution here because a StarsPokerTable “IS A” OnlinePokerTable which “IS A” PokerTable. If a function expects a single PokerTable object as an argument, you can pass in an OnlinePokerTable or a further-derived StarsPokerTable instead without altering the correctness of the program.

That sort of polymorphism is something that will be hugely helpful, and we'll use it everywhere. The pattern is this: work with base classes to the extent possible, and let derived classes transparently implement specific functionality as needed. This same sort of relationship applies throughout the poker and online poker domains. In fact, the relationship is so strong that what you'll end up with (or at least, what I ended up with) is a domain which can be physically segregated into different physical components:

• generic poker "things" (living in, for example, PokerDomain.dll).
• generic online poker “things” (living in OnlinePokerDomain.dll, which depends on PokerDomain.dll).
• specific online poker "things" (living in StarsPokerDomain.dll or FullTiltPokerDomain.dll, which depend on OnlinePokerDomain.dll).

That, in turn, offers a number of programming benefits, not least of which is decomposition of the system into layers of bite-sized components (each consisting of at most a handful of classes) with strong physical boundaries. It's also a good way to structure the source code if you intend on creating a bot that can play at multiple venues.

N-Dimensional Opponent Modeling

As we slowly but surely work our way into a candidate A.I. for the poker bot, we’ll need to start picking up certain techniques and adding them to our arsenal. One of these is the idea of representing a player’s particular strategy or inclination as a point or region in N-dimensional (2D, 3D, 4D, etc) space.

Often in poker we categorize and describe our opponents based on two things:

• Selectivity. How selective an opponent is with regard to choosing hands to play. Selective opponents who play few hands are called "tight". Unselective opponents who play many hands are called "loose".
• Aggression. How much betting and/or raising (as opposed to calling and/or checking) an opponent engages in. Opponents who frequently bet and raise are aggressive; those who rarely bet and raise are passive.

We can express selectivity and aggression on a familiar, two-dimensional Cartesian coordinate system. Here, both selectivity and aggression are expressed as numbers between -1.0 and +1.0. At (1.0, 1.0) a player is maximally selective and aggressive. At (-1.0, -1.0) an opponent is minimally selective and aggressive. 

Note that the four quadrants of this coordinate system correspond to the four “player archetypes” identified in The Psychology of Poker and other books. This works well as a rough indication of an opponent's core playing style. But I like to take the above grid and add a third axis: deception.

Deceptive opponents frequently attempt to mislead their opponents with bluffs, slowplays, and other tactics. If a deceptive opponent bets, you can’t be sure whether he’s got a strong hand or nothing at all. On the other hand, many opponents are “honest”. A big bet indicates a big hand.

That allows us to express a player's (approximate) strategy as a single point in three-dimensional space. For example, a player at [0.75, 0.75, -0.75] would be described as tight, aggressive, and straightforward (non-deceptive). We would expect this player to bluff and slowplay less often than the player at [0.75, 0.75, 1.0].

Taking it a step further, we can add additional dimensions to our graph. For example, let’s add a fourth dimension: stack size. The player at [1.0, 1.0, 1.0, -1.0] is maximally aggressive, selective, deceptive when his stack size is towards the bottom of the range. The player at [0.0, 0.0, 0.0, 0.0] plays a non-descript (neither tight nor aggressive, nor loose nor passive, but somewhere in between) game of poker when his stack is roughly average. And the player at [-1.0, -1.0, -1.0, 1.0] is passive, loose, and non-deceptive when his stack size is large.

Once we get into the multi-dimensional approach, we start to see that a given player’s strategy isn’t really a point, although we may choose to express it that way for convenience; it’s a collection of points which together comprise a strategy region in n-dimensional space, where each dimension is a particular strategy facet. This region can be mathematically described, tested for intersection with other regions, and so forth. Use your imagination.

Of course, selectivity and aggression are just two possible axes. We'll use these because they're common throughout the poker literature, but understand that the meaning and quantity of axes is up to you. In fact, it's worth your while to set up a generic mechanism by which you can capture and plot arbitrary groupings of statistics. This in fact is one of the purposes of the PokerFact abstraction (in the diagram at the top of this page) and its many associated classes.

But that subject is a post in itself. Stick it in your toolbox and we’ll return to it at a later date.

Opponent Modeling with PokerTracker

PokerTracker is a commercially available online poker hand history analysis tool - that is, a tool which parses the human-readable hand histories emitted by a particular site, analyzes them, and displays various statistics about your play and the play of your opponents:

Every serious online poker player, without exception, either owns, or knows about, Poker Tracker. In fact, Poker Tracker statistics (VPIP, PTBB/100, etc) have been adopted by the poker playing community at large, and now serve as a sort of de facto terminology for advanced poker strategy. You can hardly read a major poker forum such as 2 + 2 or Pocket Fives without understanding Poker Tracker lingo:

baja15 is winning at 17.5/13.5. chargers in o7 is winning at 13.4/11. The specific numbers are not important; what is important is that both chargers and baja are almost always raising the pots they play. Seizing the initiative is crucial to winning at tight/aggressive poker. They have the table reacting to them. When they call, which is usually about 3.5% of the pots, they have a reason for doing so. If chargers or baja went back through all the hands they called a PFR with, they would be able to tell you why they did it every time. And they would never say “well, because KQ is playable in MP” or something vague like that. It would be “because I was getting the implied odds to play 33 for a set here—villain is a 10/2 nit with 100bb who only raises UTG with premium hands; I knew I’d get at least a big part of his stack if I hit my set.”

As useful as Poker Tracker is for online poker players, it’s even more useful for online poker botters. In fact, Poker Tracker is the single most useful third-party tool in the poker bot-building arsenal and here’s why:

• It serves up a relational database (with a well-documented schema) containing 90% of the opponent historical data you'll ever want or need.
• It uses a family of statistics which have the weight of countless hours of discussion behind them.
• It's a commercially supported application with one of the most active communities in poker.
• It’s allowed, and even tacitly encouraged, by all major online poker venues.

Absent Poker Tracker and similar tools, you'd be forced to parse those hand history files yourself, and to develop a statistical framework around them yourself, and to test and verify the utility of that framework yourself. Poker Tracker does all of this for you. And while the Poker Tracker statistics aren't perfect, they've got the weight of mass usage behind them, and you'll be able to find countless discussions about the ins and outs of various statistics, what they mean, how to use them to adjust strategy, and so forth.

Poker Tracker allows you to start answering questions like:

• Is opponent X a loose or a tight player? Passive or aggressive? Winning or losing?
• Where are the holes in my (or my bot’s) game? Am I playing too loosely in the small blind?
• Am I winning or losing? How many total hands have I played? What’s my average win rate?

We’ll take a look at some working source code before too long, but if you know your way around a database, leveraging Poker Tracker shouldn’t be too much of an issue. Also, Poker Tracker isn’t the only game in town. You can also take a look at Poker Academy Prospector and Hold’em Manager, just to name a couple.

The Advantage of Playing Short-Stacked

You’ll notice that one of the Decision Modules in the above diagram is called ExpertShortStackedAnalyzer. That’s my always-inventive name for a particular class which attempts to model a game of poker in which the stack sizes have been artificially shortened.

I've mentioned a couple times the advantages of creating a short-stacked bot but I don't think I've been clear enough about what those advantages are, and why they're so helpful. First of all, when I say a short-stacked bot, I really mean a short-stacked v1.0 bot. The idea should be to get something basic up and running first, before tackling deep-stacked no-limit Hold'em.

Now, let's define what a short-stacked bot is:

A short-stack bot is a bot which buys in for between 10 and 40 times the big blind, and which leaves the table when it wins much more than that.

Most online poker sites set the minimum allowed buy-in at 20 times the big blind (20xBB), or squarely in the “short-stacked sweet spot.” In No Limit Hold’em: Theory and Practice, David Sklansky explains:

Playing a Short Stack

Since most of the money gets bet early in the hand, and betting all-in necessitates a showdown, short stacks should concentrate on playing premium hands and generally avoid bluffing. Play tightly preflop and try to get your money in when you likely have the edge against the hands your opponents might have.

Indeed, a totally algorithmic strategy of playing only premium preflop hands (say AA-77, AK and AQ) will beat almost any no limit game when you have a short stack of approximately twenty times the big blind. Ed suggests just such a strategy to beginners, in more detail, in his book, Getting Started in Hold’em.

That’s not to say that such a strategy is optimal. For expert players, playing short routinely is a sure way to slash win rates. But it is remarkable that a robot playing a short stack can beat even relatively tough no limit games. Don’t underestimate the intrinsic advantages of playing a short stack.

He goes on to give two reasons why the short-stack has an intrinsic advantage:

• It’s often correct for deep stacks to play loosely against other deep stacks, but incorrect for them to do so against short stacks. So the short stack is often up against weaker-than-average hands versus his range.
• Once the short stack is all-in, opponents will keep betting, forcing players to fold. These folds benefit the short stack mathematically, the moreso because his additional risk is zero.

To which I’d add:

• The average player is 100% clueless about how to play, and how to play against, a short stack - whether in a cash game or tourney situation.

Another reason the short-stacked approach is so powerful is that commitment decisions become much easier as the stack sizes dwindle. In Professional No-Limit Hold’em, Matt Flynn, Sunny Mehta, and Ed Miller write:

When the effective stack sizes are 50BB [or less], you can easily create favorable SPRs for top pair and overpair hands with straightforward raising tactics.

If you're not familiar with the terminology, an SPR (stack-to-pot ratio) is simply the ratio of your stack (or technically, the smaller stack) to the total size of the pot on the flop. It's a numeric expression of risk-vs.-reward at the nexus of the hand which, among other things, tells you when you welcome aggression and are willing or even eager to get all of your money in. The beauty of playing short-stacked is that the mix hands which are correct to play when you're short-stacked always produce favorable SPRs for getting your money all-in.

From a programming perspective, that often (not always, but more often than you’d think) reduces the flop decision to a binary one:

• YES, I’m somehow going to get my money all-in here, probably with a straightforward bet or raise.
• NO, I’m not willing to commit. I fold.

That’s huge, in terms of cutting into that complexity I was talking about. But it doesn’t end there. Playing short-stacked desaturates the game in a number of ways that simplify your job as the programmer:

• A short-stacked bot rarely has to consider any actions beyond the flop. This has the effect of turning a 4-street game (preflop, flop, turn, river) into a 2-street game (preflop, flop).
• Preflop play, while subtle at times, is fairly easy to abstract into a set of rules. This is somewhat true even in deep-stacked play, but it's especially true in short-stacked play.
• Flop play, the crux of the hand, can be simplified through the application of SPRs and some basic commitment mathematics (as I explained above).
• Any mistakes or weaknesses in the bot can only be punished in small increments.
• As Sklansky notes, it’s usually incorrect for a short stack to bluff. That means you don’t have to implement game-theoretic bluffing, and you don’t have to implement opportunistic bluffing. In fact, beyond an occasional continuation bet, which will put you all-in anyway, you don’t have to think about bluffing at all.
• Short stacks don’t care or worry about implied odds. Estimating implied odds is one of those fuzzy areas in which humans prosper and robots perish.
• Buying in short is appropriate when the other players at the table are better than you, which will often be the case with a home-grown bot.

So while I don’t suggest that you rule out the idea of creating a bot that can play deep-stacked, I do think the first step on the road to that goal is creating a bot which is capable of playing short-stacked.

The Importance of Generic Code

If you’ve read this far, then you’re presumably pretty interested in this subject. So a friendly word of advice.

One of the reasons many bots fail is because they're designed and developed in a vacuum. Best practices are abandoned in favor of the quick and dirty approach. Code that would never fly in a commercial production environment is created, tweaked, reinforced, cut-and-pasted. The result: a useless, hacked-together mess of spaghetti code, or what's known as a Big Ball of Mud.

In a project of this complexity, avoiding this sort of thing has to be a fundamental requirement from Day One. Even if you're not successful in building the bot, keep your options open. Maybe some of the code can be usefully applied to another project. At all costs, avoid burning a year or more of your time producing a glut of bad "poker botting" code. Keep it clean, keep it reusable.

One of the better ways to lift your code out of the trenches and increase its utility and longevity is to rely on a tried-and-true software development best practice:

Botting Rule #2: Code the domain, not the problem.

When we apply this rule to the domain of poker, what we get is a code base in which the classes and other abstractions have a strong correlation with meaningful nouns and verbs:

• poker player
• poker table
• online poker table
• poker game
• poker hand
• hand history
• poker action (check, raise, bet, fold, etc)

Whether you’re building a bot, a commercial poker game, or a simulation tool, you’re probably going to be interested in the concept of a poker table. That’s a core abstraction which establishes an ordering amongst players. Similarly, you’ll want some notion of what it means to be a poker player. That's another core abstraction. And so on, until you end up with a group of reusable classes similar to those shown in the diagram presented above.

Taken together, and with the addition of a motive intelligence, these classes comprise a poker bot. But they’re also useful in creating a hand history analyzer. A game simulator. A multi-tabling helper tool. An online poker heads up display (HUD). And so forth. If you take the proper care to model things this way, you’ll find that your code has useful applications quite aside from botting. That, in turn, further increases the utility of the work you’re performing, boosting the “reward” half of the risk/reward equation we talked about in the Introduction.

Ignoring this bit of advice cost me six months (easily) of development time in the creation of the above bot.

Conclusion

We’ve covered a lot of material in this post, including:

• Why building a poker bot is an ideal pet project.
• A light introduction to advanced poker A.I.
• A candidate architecture for a generic "pluggable intelligence" poker bot A.I.
• How to use an iterative, incremental approach to apportion the bot into bite-sized pieces.
• Some working advice and best practices for modeling the poker domain.
• N-Dimensional Opponent Modeling.
• Poker Tracker, and its value to the player/botter.
• Why short-stacked bots are so tough to beat.
• Avoiding the poker bot “Ball of Mud” dilemma.

In Part Two of this series, I promised you some working code for interrogating the poker client for various game data. That code has been rolled into its own dedicated post, and in fact, this is how the series will be structured from here on out: long, rambling higher-level posts interspersed with short, targeted, source-code rich posts solving specific problems.

I’d also like to apologize for the delay in responding to some of your questions and comments. These are all excellent topics for discussion and I’d like to address them fully, in dedicated posts. But however we structure it, the poker botting series will continue in exhaustive detail. If you’ve enjoyed the first three parts of the series, and you haven’t already done so, please subscribe in a news reader or over email. And as always, thank you for your time.

Until next time…

I’m going to go out on a limb today and try to paint a picture of where some of this cool and crazy cloud-based infrastructure may be going. While none of what I will write about is idle speculation, it is based on just a few data points, and may be totally off base. However, I do get to talk to plenty of entrepreneurs and developers on a daily basis, and I am starting to see a very interesting pattern emerge.

The existing state of the art in cloud-based architectures takes the shape of an application running in the cloud, calling upon services running within and provided by the operator of the cloud. There are any number of great examples of this type of architecture. Doug Kaye at IT Conversations built and documented his implementation over a year ago. Earlier today, Don MacAskill of SmugMug send me a link to his new post, SkyNet Lives (aka EC2 @ SmugMug). In that article, Don provides a detailed review of SmugMug’s use of Amazon EC2 and S3 to implement a dynamic, highly scalable system which simultaneously minimizes response time and cost by optimizing the number of EC2 instances.

As I said, I am starting to see something which goes beyond this in a subtle yet important way. Developers are now building services in the cloud for other developers, with the understanding that important (and perhaps primary) consumers of the service will also be resident within the same cloud.

I’m going to call this the CSA, or Cloud Service Architecture.

Applications communicating with each other inside of the Amazon cloud enjoy some important benefits. They get high-bandwidth, low-latency communication, at little or no cost. They inherit all of the other attributes of cloud-based applications such as on-demand scalability, fault tolerance, cloud-wide network security, and cost efficiency. Applications running in loosely coupled fashion within the cloud can share data using SQS, S3, or other communication protocols of their choosing.

Right now, I see that forward-looking companies are starting to build components which fit into the CSA. On the database side, we have Vertica for the Cloud and MySQL Enterprise for EC2. On the media side, there’s Cruxy’s MuxCloud, IntrIdea’s MediaPlug, and Wowza Media Server Pro for Amazon EC2. I’m sure that there are others that I don’t know about.

So who’s calling these services from other EC2 instances within the cloud? Here are my first two data points (that’s enough to draw a trend line, right?):

1. I had breakfast with the CEO of Sonian yesterday. He told me that they are now using the Vertica product to help them store, index, and retrieve massive amounts of data (more info can be found in their case study).
2. Earlier this year I paid a visit to VisualCV in Reston, Virginia. They use MediaPlug to support uploading and processing of a variety of types of images and videos.

My sense is that this is the start of something big. Web services made it possible to cross organizational boundaries with a simple HTTP request. Now, running within the cloud makes it possible to do this with minimal network latency.

As individual developers learn more about cloud computing, they will naturally look for some very high-level components up and running within the cloud. Over time I am sure that there will be a need for more sophisticated tracking and billing mechanisms, key management, a catalog of services, and other facilities that we can’t even envision just yet. As always, we love to get this feedback from you, so let us know what you need.

I’m sure that there are some other CSA-style applications running in the Amazon cloud now. If you’ve built one, post a comment!

– Jeff;

The folks at Intridea have been coding up a storm!

In time for this week’s Railsconf, they are rolling out a hosted, on-demand version of their popular Scalr tool, a new release of their MediaPlug media server appliance, and the Acts as Community social network.

Scalr gives enterprise IT professionals the ability to quickly and easily set up and run EC2-powered server farms. Once such a farm has been set up, Scalr monitors and maintains it, with automatic scaling, failover, and redundancy. Scaling is based on load averages, with automatic instantiation of new instances of the proper type once the aggregate load average reaches a configurable threshold. If an instance crashes, Scalr replaces it with a new instance of the proper type. The hosted version is available for $50 per month; the open source version can be downloaded here.

MediaPlug is an EC2-powered media server appliance, packaged and sold as an AMI (Amazon Machine Image). It supports transcoding, uploading, and storage of  images, audio files, and video files in a number of popular formats. There's a MediaPlug web service for direct integration into backend code, along with a JavaScript library for the front-end. MediaPlug is priced at double the cost of a Small EC2 instance, or approximately $150 per month.

Finally, Acts As Community, is a social network for Ruby developers. Of course, the site runs on Amazon EC2 and is managed by Scalr. The site includes forums and blogs for open source projects hosted at GitHub; user group creation and management tools, personal profiles, and a number of community features such as forums, questions and answers, media sharing, and code sharing.

– Jeff;

Seattle-based Napera Networks has built a really interesting piece of hardware. The Napera N24 Appliance automatically enforces health and identity rules for up to 200 computers, preventing rogue or unpatched machines from accessing a corporate network in an uncontrolled fashion.

The device basically protects access points in conference rooms, sales offices and wireless networks from indiscriminate usage. Napera uses the Microsoft NAP (Network Access Protection) protocol to enforce a fully configurable set of health and safety rules. Logging and monitoring features provide auditing and other historical information.

At this point you are probably thinking "Uh, Jeff, you work for Amazon.com and you usually talk about software on this blog. What's going on here?"

So here’s where it gets really cool. The Napera N24 appliance securely stores
all of its long-term logging and auditing data in Amazon S3. This way,
there’s no limit to how much can be logged or how long it is retained, unlike
a more traditional design which would retain some logs within the appliance
or require local servers. Further, the administrative
interface (MyNapera.com) leverages Amazon EC2. Ultimately MyNapera.com will
be accessible remotely, making the Napera appliance a great solution for
a company with a number of geographically distributed field offices.

Enterprise solutions to this sort of problem are usually very complex
and expensive, requiring multiple servers. Using AWS has enabled Napera to
build a product for the budget of smaller companies that is incredibly easy
to use and quick to install.

Oh yeah, one last thing. The company is based in Mercer Island (near Seattle). Per their jobs page, they are looking for a web developer and some summer interns with CS or IT experience.

– Jeff;

Introduction

Last week, I posted about how I built a working, real-money online poker bot. This week, we'll discuss how to get started building your own bot, address some specific bot-building techniques, talk about the woeful state of online poker user privacy, respond to a few reader questions and comments, and close out with some recommended reading.

This series is both a detailed manual on how to build and make money with an online poker bot, and a story of how two guys from the Dallas poker underground managed to realize such a bot in practice.

We'll be discussing everything necessary to build what's quickly becoming the de facto standard in online poker botting: a sophisticated parallel processing rig incorporating collusion AI, massive hand history databases, and real-time information flow to dominate the opposition. This is where the world is headed:

NOTE: While I believe building and running an online poker bot to be neither cheating nor unethical, building a ring of such bots is a different story. As a long time poker-player, I absolutely don’t condone cheating people out of their money by colluding in online poker or any other game. We're going to discuss things like bot collusion not so we can (all of us) go out and build colluding bot rings, but because these are the techniques people are using right now, today and tomorrow, to gain an unfair edge in online poker.

In order to build, or learn how to defeat, such a contraption, we're going to have to cover a lot of ground:

• Artificial Intelligence. Neural nets, genetic algorithms, rules engines, decision trees.
• Poker Strategy. Not the stuff you see on TV. Expert EV-driven poker strategy as it exists today, at the tables, in books, and on the forums.
• Input Simulation. How to generate an appropriately timed and positioned stream of mouse moves, clicks, and keyboard input.
• Operating Systems. Code injection, API hooking, kernel objects, multi-threading, DLLs. In general: forcing Windows to do what you want.
• Reverse Engineering. How to reverse engineer poker client applications and the network data streams they rely on.
• Probability and Statistics. Bayes Theorem, probability distributions, confidence intervals, and other goodies.

If you’re new to poker, and particularly if you’re new to programming, this will be a little bit of a trial by fire. Building a bot is basically the programmer equivalent of joining a Fight Club.

A guy who came to Fight Club for the first time, his ass was a wad of cookie dough. After a few weeks, he was carved out of wood.

Sort of. Read on.

Why are so many people interested in building bots?

Last weekend I got a phone call from a friend (a guy I’ll have more to say about later), informing me that How I Built a Working Poker Bot, Part 1 had hit the front page of Digg, reddit, del.icio.us, and other sites, and that Coding the Wheel was getting enough traffic to occasionally throttle the server.

Since then, readers have posted upwards of a thousand comments to the story, here and around the net, and I've gotten a couple hundred emails ranging from technical diatribes to job offers to good old-fashioned hate mail.

The response has been unexpected. Thanks to all who took the time to read the first article, and to those who've joined the discussion so far. In hindsight it seems obvious, but many, many people have a vested interest in building, or discouraging the building of, software tools which emulate human behavior. It goes way beyond online poker, then again, in online poker the profit motive is compelling:

A successful online poker bot is worth hundreds of thousands of dollars even if all your bot does is break even. Sensationalistic? Maybe, but it's grounded in the very dull, non-sensational mathematics of rakeback and player promotions, which I've outlined below, and which we'll revisit in future posts.

Botting Rule #232: Wherever mouse clicks or other human-computer interactions are worth money, hordes of people will try to write software to simulate those interactions.

Why online poker clients are basically spyware

I want to preface this by saying: the online poker sites aren’t evil. They’re trying to protect their business, and that’s perfectly normal. But I think they’ve strayed from the beaten path when it comes to respecting the privacy of their users.

Many online poker players have no idea that major poker sites silently install a browser plugin which has access to every page you visit in your browser, and potentially every piece of information on that page (the text of private emails; credit card numbers; user login information; etc):

Many online poker players have no idea that poker clients can and will snoop around on your system, potentially viewing sensitive data:

• By examining your list of running processes
• By reading body and titlebar text from every window you have open
• By taking occasional screenshots
• By snooping around on your file system, and in the system registry
• By doing who knows what else, since there’s zero oversight.

This is the definition of spyware. These "safeguards" constitute a basic invasion of your digital privacy and you should be angry about it. It doesn’t matter whether the sites actually collect, pay attention to, store, or transmit that information. It doesn’t matter if they do it with the best of intentions. It doesn’t matter if they tell you they’re going to do it in the EULA. They quite simply have no business doing it at all, and if they’re going to do it, they should be doing a better job.

In other words, online poker is really the worst of both worlds: extreme invasion of privacy unheard of outside the realm of spyware, and zilch to show for it - zero effective safeguards against bots or other supposedly malicious software.

And yes: it’s true that many sites will recognize well-known botting software such as WinHoldem. But they currently do very little in the way of preventing home-grown bots.

Botting Rule #472: The biggest advantage of a poker bot is its obscurity.

Using Spy++ or Winspector to get basic information from the poker client

Spy++ and Winspector are useful window analysis and debugging tools which allow you to view window properties and messages for any window in the system, including poker client windows:

If you take a close look at the above screen shot, you'll see that you'll see that Spy++ has extracted the handle (HWND), caption, class, style, location, and other properties of a Poker Stars table window. We can also drill down and find its child windows, parent window (if any), window class (PokerStarsTableFrameClass, in case you're curious) and other assorted information.

We can also use Spy++ to snoop on the exact messages received by a given window:

Spy++ is included in all versions of Visual Studio, as part of an MSDN subscription. Winspector has a larger feature set than Spy++ and is is available as a free download from http://www.windows-spy.com/.

Later, we'll be using this tool to help us figure out what the poker client is doing, beneath the hood. It's also an extremely useful all-purpose window debugging tool, and an important part of a bot developer's (or any Windows developer's) toolkit.

Why a bot doesn’t have to win, in order to win

I've stated repeatedly that a poker bot only needs to break even in order to generate a profit. Here's how it works, for those who aren't familiar with rakeback or other online poker promotions:

• IF you play X hands of online poker within a certain amount of time (such as a calendar year).
• THEN the poker site gives you Y amount of money, or "player points" which are worth money.

That’s a simplified, but otherwise accurate, description. And listen. Y can be very large, even “life-changingly” large. Let’s look at some actual figures.

Consider that it's fairly standard for a (human) player to play 8 or even 12 tables simultaneously.

Furthermore, in online poker you're generally getting between 50 and 70 hands per table per hour. Let's call it 55, just to be conservative.

So that's 10 tables @ 55 hands per table per hour, for 10 hours a day, 6 days a week, 50 weeks a year.

If you do the math, you’ll find it comes out to around 1.65 million hands per year. If your bot is playing anything other than the microlimits, this is easily enough to qualify you for (just as an example) the Poker Stars Supernova Elite promotion, which when properly liquidated is worth around seventy or eighty thousand dollars the year you achieve it, and two or three times that amount the following year (because of the multiplier effect of FPP bonuses).

Similarly, other sites offer rakeback promotions (~30% rakeback is fairly standard) along with assorted perks. If your bot is playing break-even poker, then your rakeback - quite substantial, as you move up from the microlimits - is pure profit. Ironically, your profit would shrink to zero if the site stopped collecting rake! (Update: a few readers have questioned this statement, claiming that a break-even bot with rake equates to a winning bot without rake. This is true, but the statement was tongue in cheek. It's ironic that a break-even bot ultimately ends up making money because of, rather than in spite of, the rake.) The thing that more than any other makes games tough to beat - the rake - makes it possible for your bot to turn a healthy profit. And that, as a long-time online poker player, makes me smile.

Botting Rule #1274: Many online poker sites would love to allow bots, if only their users would let them.
Botting Rule #47: Online poker players fear bots all out of proportion to the average bot’s ability to win in competitive poker.

How to simulate human input (an overview)

In next week’s post we’ll cover this topic in detail, but for now: in order for your bot to be able to perform actions such as betting, raising, or folding, it needs to know how to talk to the poker client. There are at least three basic mechanisms you can use:

• (Most difficult) Reverse-engineer the network protocol and communicate directly with the poker server.
• (Somewhat difficult) Hook into the poker client just beneath the user-interface. In this scenario, you don’t simulate mouse clicks; you figure out what internal function(s) handle those mouse clicks, and call those functions directly.
• (Easy) Simulate user input - mouse clicks, keyboard input, whatever - at the operating system or even the driver level.

The first two techniques are mostly black magic. This third technique - user input simulation - can be further broken down into:

• Using the SendInput API.
• Directly generating and posting WM_MOUSEMOVE, WM_LBUTTONDOWN, and other messages to the poker window.
• Using the (deprecated) keybd_event and mouse_event APIs.
• Writing a custom mouse or keyboard driver.
• Possibly other techniques..

And regardless of which method you use, you'll want to make it realistic:

• By incorporating subtle timing randomizations.
• By creating realistic mouse-movement trails.
• By occasionally clicking and interacting with unrelated windows.

You’ll find that the SendInput method offered the best tradeoff between power and ease of use, but this is a complex topic. I mean: the interactions between user input events and the various Windows subsystems are complex, and will require tweaking to get right. But the code to generate those interactions is fairly simple.

Why poker bots will soon be accepted as some of the strongest players in the world

In the first-ever Man vs. Machine Poker Championship, a University of Alberta research team lead by Dr. Jonathan Schaeffer (the guy behind Chinook, the program that effectively solved the game of checkers)…

…piitted the Polaris poker bot against famous poker professionals, Phil “the Unabomber” Laak and Ali Eslami.

Laak/Eslami won the match with 2 wins, 1 draw, and 1 loss. That's not a hugely convincing margin, and in fact, as the University of Alberta website points out:

The match was a success in many ways. Polaris proved that it was able to compete with some of the best poker players in the world. In fact, the 2-1-1 record of the humans is a little misleading. The actual difference in monetary outcome was just $395 which is a very small amount. The format of the match did a great deal to reduce the large variance in the game of poker, but it does not remove it all. The $395 sum could be as few as one or two hands where Polaris decided to fold when the human who got the same cards decided to continue. For this reason, a future match should prove particularly interesting, as the bot continues to develop in strength.

As for Phil Laak, he had this to say about Polaris, and about poker bots in general:

We’re already at the point where artificial intelligence crushes players that are unsophisticated, beats handily intermediate players, and loses small or wins small against savvy opponents… For Round 1, I’d say the bots have it.

Polaris's performance is reminiscent of the 1996 match in which world chess champion Gary Kasparov fended off the Deep Blue supercomputer, for the last time.

The IBM team responsible for Deep Blue made a few tweaks, and one year later, in 1997, Deep Blue came back and won a historic match, 3.5 to 2.5 (that's two wins, one loss, and three draws). Kasparov's allegations of unfair software-coaching practices were mostly ignored, and IBM never granted him a rematch. That caused some people to question the authenticity of Deep Blue's victory, for almost a decade.

Until 2006, when another chess program, Deep Fritz, handily beat world champion Vladimir Kramnik. The crucial difference between Deep Blue and Deep Fritz:

• Deep Blue ran on specially designed supercomputer-grade hardware
• Deep Fritz ran on a workstation PC with two Intel Core 2 Duo CPUs

For now I just want to make the point that intelligent, commercial-quality poker bots are a reality in the low and middle limits, and have been for a few years.

• MSNBC: Are poker ‘bots’ raking online pots?
• Wired: On the Internet, Nobody Knows You’re a Bot
• Coding Horror: The Rise of the Poker Bots
• ThisIsMoney.co.uk: We put poker bots to the test…

Why DLL injection is so powerful

Several readers have stated that techniques like DLL injection are either a) overkill or b) likely to get you in trouble with the online poker sites. So I’d like to address this issue in a little more detail.

First, here's a video which demonstrates a simple, harmless use of DLL injection in practice: overwriting the Poker Stars cashier balance with a balance dreamt up by the user.

• Inject the DLL
• Subclass the Cashier window
• Detect when the Cashier window is invoked
• Override WM_PAINT and display a “fake” balance to the user

First, I recommended it earlier, and I’m going to recommend it again: Jeffrey Richter’s excellent book on advanced Windows API development: Windows via C/C++. Buy it. Borrow it. Steal it if you must. Most of the information you'll find inside is available somewhere on the Internet, but you'll spend hours upon hours tracking it down. And nobody - nobody - speaks on this subject with Richter's authority.

Here are some thing to keep in mind when deciding whether or not DLL injection is for you:

• DLL injection isn't a hack. It's a formal capability offered by the Windows API, without which hundreds of legitimate applications (ranging from computer-based training apps to instant messaging applications) would stop working.
• There are at least 6 ways to inject a DLL (or binary) code into another process on Windows operating systems: 1) via the Registry 2) using Windows hooks 3) as a Debugger 4) with Remote Threads 5) by creating a Trojan DLL 6) CreateProcess/WriteProcessMemory.
• DLL injection isn’t particularly difficult to implement, and it’s not a poorly-documented, error-prone procedure. It more or less just works.
• A given poker client can't simply declare that all injected DLLs are evil. The operating system, as well as legitimate third-party applications, can both cause DLLs to be mapped into a process's address space. If Poker Stars, World of Warcraft, or whatever other application were to simply shut down whenever this happened, they'd be unusable.
• While it’s possible to detect when DLLs are injected into the process’s address space, it’s not possible to determine whether a given DLL is innocent or malicious. Sure, you can look for the DLL's name, but the name can be changed, with every invocation if need be. You can create some sort of checksum to try to identify malicious DLLs at the binary level. Again, easy to get around by creating a self-modifying .EXE.

That said, is it possible to create a bot without DLL injection? Absolutely. But if you can't get your code to run in the poker client's process, your powers are somewhat limited. For example, you won't be able to subclass poker client application windows. You won't be able to query various process data structures. You won't be able to intercept Windows messages intended for the poker client. And so forth.

Why screen-scraping is a bad idea

A lot of people have argued that some of the screen-scraping techniques I mentioned - such as pixel-testing for hole cards - are difficult to code, error-prone, and tedious to maintain. I couldn’t agree more. Screen scraping is a last resort. The bot’s Input Component needs to be structured such that it falls back on screen-scraping techniques only when all other options are exhausted.

That said, a mature bot capable of playing at multiple sites will probably require some sort of screen-scraping capability. The trick is to create a generic screen-scraping mechanism driven by XML or some other easily-edited format. That way, when some minor aspect of the poker client UI changes, all you have to do is update the “schema” for that particular online poker venue, without rebuilding the poker bot executables.

This will be the subject of (you guessed it) a future article.

Recommended Reading

This is by no means a complete list of recommended programming and/or poker books. However, all the books on this are a) relevant to building a poker bot and b) contain information which isn't readily available on the Internet, at least not without a good deal of sleuthing. I own a dog-eared, coffee-splattered copy of every book on this list, and have read most of them multiple times. Note: the programming books are coached from a C++ and Windows perspective but the techniques can be ported to other environments/languages.

Programming books:

• Windows via C/C++. Jeffrey Richter. Everything you ever wanted to know about the Windows API. Should be required reading for all Windows developers, and for anybody working with .NET, as it addresses many of the API constructs on which .NET is built.
• Exploiting Online Games: Cheating Massively Distributed Systems (Addison-Wesley Software Security Series). Greg Hoglund and Gary McGraw. If you actually plan on building an online poker bot, this is the book for you. Discusses many of the same techniques discussed in this series, but with a focus on World of Warcraft and other MMORPGs. All of the techniques discussed apply to poker, however.
• Exploiting Software: How to Break Code (Addison-Wesley Software Security Series). Greg Hoglund and Gary McGraw. Companion volume to the above, with an emphasis on local applications (e.g., the poker client application running on your machine).

Poker books:

• The Theory of Poker. David Sklansky. Quite possibly the most influential book in poker, written by the father of modern poker theory, David Sklansky. The value of this book is that it explains poker theory in a way that’s universally applicable to all poker games. Required reading for all poker players and poker bot programmers.
• The Mathematics of Poker. Bill Chen and Jerrod Ankenman. Difficult but extremely rewarding exposition of poker mathematics. In order to program a fluent poker AI you'll want to understand a lot, though not necessarily all, of this material.
• Professional No-Limit Hold ‘em: Volume I. Matt Flynn, Sunny Mehta, and Ed Miller. In my opinion this is the best book on no-limit Hold’em ever written, and the notion of planning hands around commitment and SPR squares very well with the poker bot AI we’ll get into later.
• No Limit Hold’em: Theory and Practice. David Sklansky and Ed Miller. Another excellent no-limit Hold’em book.
• Gambling Theory and Other Topics. Mason Malmuth. A somewhat dry, rarely-read masterpiece, with some non-trivial mathematics. Not everything here is directly applicable to playing, or building software for, online poker, but it’s all useful.
• Getting Started in Hold ‘em. Ed Miller. Introductory text. There are many good introductions to poker and to Texas Hold’em; this is one of the better ones, and introduces a short-stacked strategy that will be useful in deriving a competitive bot AI for real-money games, down the road.

Conclusion

We took a break from the source code in this installment as I don’t want to alienate too quickly the readers who’ve subscribed in the past week. In the next installment, we’re going to have to plunge right back in, beginning with an in-depth introduction to the architecture of the bot’s AI component, expressed in C++, C#, or the language of your choice, and concluding with specific code you can use to simulate a realistic stream of human input. We’ll also perform explorative surgery on various online poker clients, exposing their inner resources and code structures to the light of day. Last but not least, we’ll answer the question of: how do I snoop on the poker client’s log or hand history file in real time?

As always, thanks for reading, and stay tuned for the next installment in the series. Until then…

XML Hacker M. David Peterson has put together a really interesting article.

As part of his work at 3rd and Urban, he has implemented redundant, fault-tolerant, read-write disk storage on Amazon EC2 using a number of open source tools and applications including LVM, DRBD, NFS, Heartbeat, and VTUN.

Mark notes that "the primary focus of this paper is to present both a detailed overview
as well as a working code base that will enable you to begin designing,
building, testing, and deploying your EC2-based applications using a
generalized persistent storage foundation, doing so today in both lieu
of and in preparation for release of Amazon Web Services offering in
this same space."

The article provides complete implementation details and links to source code for the scripts that Mark developed.

You can read the article, and you can also follow progress via the discussion group.

– Jeff;

• foxgame: index
• Install yum with rpm on CentOS
• Install yum - OpenVZ Wiki
• WordPress › OpenID Provider for WordPress MU « WordPress Plugins
• VM Planet | VMware Images
• RightScale: Launch scalable Amazon EC2 instances
• S3Browse - Amazon S3 Web Frontend
• elastic8 | Elastic computing
• Super Flexible File Synchronizer
• Lumen Systems
• Dynamic Host Information Services
• Amazon Web Services @ Amazon.com
• Amazon Web Services Developer Connection : Resource Center
• Web Log Analysis for Amazon S3 - S3STAT
• Apatar - Open Source Data Integration & ETL
• Data Bucket Pro
• Compactador de Batallas OGame - MecaHost

Lanzamiento de la primera versión oficial de OpenSolaris, el sistema operativo libre de Sun basado en el kernel de Solaris. OpenSolaris apunta más a pequeñas empresas y desarrolladores finales que a las grandes empresas que típicamente han hecho uso de Solaris, aspectos como su propio “apt-get” (pkg) y el planteamiento de desarrollo “en comunidad” le acercan más a GNU/Linux. Toda la información y la descarga de OpenSolaris en el sitio oficial.

Pero además del lanzamiento del nuevo sistema, Sun ha anunciado el comienzo de una estrategia para integrar sus productos con los Web services de Amazon. Así, se podrá utilizar OpenSolaris sin necesidad de tener un servidor para ejecutarlo, al estar disponible en Amazon EC2, al igual que ya lo están otros sistemas como Red Hat. La apuesta no queda ahí, desde ayer Sun ofrece soporte para MySQL integrado con EC2, un escenario hasta hace poco peliagudo pero cava vez más interesante, toda vez que Amazon añadió persistencia a su servicio (amazonwebservices.com).

Los pasos dados por Sun en las plataformas como servicio parecen otorgarle un papel secundario, cediendo el protagonismo a nuevos actores como Joyent o Amazon, en lugar de apostar por soluciones propias al 100%. Tal como está el mercado, probablemente sea la posición que más ase ajusta a la realidad de Sun, ser el proveedor/socio tecnológico, delegando “la nube” en terceros.

Más información: MySql.com, eWeek, GigaOM.

Relacionados: Sun compra MySQL, Comienza la liberalización de OpenSolaris.

[download#2]

http://tantannoodles.com/toolkit/wordpress-s3/

• SmartFTP - Cliente FTP
• net2ftp - a web based FTP client
• FlashFXP by iniCom Networks - FTP Client Software
• InfiniteBits.info :: Web and FTP access to Amazon S3 :: FTP Server for S3
• Google Analytics

• Using Amazon S3 as an Image Hosting Service Coding Horror:
• JungleDisk - Reliable online storage powered by Amazon S3 ™ - Jungle Disk
• S3 Firefox Organizer
• http://gigaom.com/2006/07/13/startups-embracing-amazon-s3/
• Amazon.com Sign In
• The Lightbox Clones Matrix « planetOzh
• http://opendns.com/
• aNieto2K | Desarrollo web, Wordpress, y alguna cosilla más
• CodePlex.SpaceBlock - Home
• CodePlex - Open Source Project Hosting
• Online Status Indicator | for AIM, ICQ, IRC, Jabber, MSN, Skype and Yahoo
• Announcing Bungee Application Server « Bungee Connect Developer Network
• Bungee Grid now running on Amazon EC2 and accessible to Bungee Developers. « Bungee Connect Developer Network
• WideLens™ Calendar :: Home
• Bungee Connect :: Platform-as-a-Service
• Ayuda WordPress en Español