A while into one of my previous positions, I was reviewing some of the code behind our main web site, all hand coded by one of my colleagues.
As I came to the contact us form, I noticed the typical spam blocking technique of asking the user to do a small mathematical equation... you know, the normal "What's 2 + 5?" type question.
Interestingly, the form's submit button was not enabled to begin with. As soon as the equation was answered correctly, however, the button became active and the user could submit the form.
Taking a quick look at the code in the browser, it was obvious that this form was entirely using JS to 'enforce' the spam protection. That is to say, JS was required to enable the submit button, and there was no actual server side validation of the spam protection.
Knowing that this is really not the way to do this at all, I asked the developer about it. His answer was that "of course it works... we haven't been getting spammed, have we?"
The answer reminded me very much of Homer Simpson buying Lisa's tiger repelling rock.
Anyway, I tried to make headway with the developer, trying to discuss bots not usually running JS, and so on, but it made no sense to them... as far as they were concerned, it worked as expected.
I finally realised I was just hitting my head against a brick wall and continued with the rest of my work. It was true, we weren't being spammed, and if worse came to worst I could resolve it quickly myself if ever we began to be spammed.
The developer left us not long after, but it was before another colleague and I discovered Google's Skipfish. Without thinking too critically (though we were sure to make sure we weren't going to try anything destructive!) we set it on the site. Things were going really well, until it found our contact us form... and hit it with 100s of different tests. That was the day that we found out that the contact us form also did something else interesting: it sent the email to about a dozen different staff members. By the time it was stopped, I think they each got around 1000 emails, which was really embarrassing.
But at least, I guess, we weren't attacked by any tigers.
Monday, December 17, 2012
Wednesday, December 5, 2012
Medium vs Regular?
"Run Lola Run" is an awesome movie. It's one of very few films I've seen more than once at the cinema. It's definitely the only one I've seen 5 times in one week.
The funny thing is, of all of that movie going, only one thing sticks out in my mind: my "discussion" with the owner of the place.
Firstly, understand that this was not at some mega-chain-cinemaplex... it was a small, privately run place. The owner was known to be quite self righteous and the only being more important than her was her little dog, that used to walk around the place and snarl at anyone it didn't like. Secondly, being in Australia, know that there are a certain number of people that dislike any form of Americanisms.
Anyway, it was the middle of the movie and I was sorely in need of a refreshment. I briskly walked to the counter and asked the person behind it for "a regular Coke". Note that the cashier behind the counter was not the owner... the owner was at the other side of the foyer at this time.
Apparently my request got the owner riled up, as not a split second after I asked for this "regular Coke", she called out (no doubt over the head of the dog but still down her own nose) "Regular? What size is regular? In Australia, we only have small, medium and large... I have no idea what size you are asking for".
A lot of the time I ignore this kind of useless banter. Other times, I bite. This was one of those times.
I looked at her in disbelief and said "firstly, it doesn't matter if you know what size it is, as I wasn't asking you for it. Secondly, regular would be whatever size is most regularly ordered here... I also don't know what size regular is, as I don't know what size is ordered most, but that's the size I want. I want whatever size is most regularly ordered."
The short "fine then" meant that I was free to continue the transaction apparently.
The best part of the whole interaction was the look on the cashier's face as they tried to stop laughing at their boss.
Every now and again I am reminded of this discussion, generally when I hear one of two phrases... both of which are relatively common in the software industry. The first is any reference to the Robustness Principle ("Be conservative in what you send, liberal in what you accept"). The second thing I hear a lot that reminds me of this incident, of course, is "Shut up and take my money".
Anyway, it only slightly soured the night. Thankfully, I had the rest of "Run Lola Run" to make up for it...
The funny thing is, of all of that movie going, only one thing sticks out in my mind: my "discussion" with the owner of the place.
Firstly, understand that this was not at some mega-chain-cinemaplex... it was a small, privately run place. The owner was known to be quite self righteous and the only being more important than her was her little dog, that used to walk around the place and snarl at anyone it didn't like. Secondly, being in Australia, know that there are a certain number of people that dislike any form of Americanisms.
Anyway, it was the middle of the movie and I was sorely in need of a refreshment. I briskly walked to the counter and asked the person behind it for "a regular Coke". Note that the cashier behind the counter was not the owner... the owner was at the other side of the foyer at this time.
Apparently my request got the owner riled up, as not a split second after I asked for this "regular Coke", she called out (no doubt over the head of the dog but still down her own nose) "Regular? What size is regular? In Australia, we only have small, medium and large... I have no idea what size you are asking for".
A lot of the time I ignore this kind of useless banter. Other times, I bite. This was one of those times.
I looked at her in disbelief and said "firstly, it doesn't matter if you know what size it is, as I wasn't asking you for it. Secondly, regular would be whatever size is most regularly ordered here... I also don't know what size regular is, as I don't know what size is ordered most, but that's the size I want. I want whatever size is most regularly ordered."
The short "fine then" meant that I was free to continue the transaction apparently.
The best part of the whole interaction was the look on the cashier's face as they tried to stop laughing at their boss.
Every now and again I am reminded of this discussion, generally when I hear one of two phrases... both of which are relatively common in the software industry. The first is any reference to the Robustness Principle ("Be conservative in what you send, liberal in what you accept"). The second thing I hear a lot that reminds me of this incident, of course, is "Shut up and take my money".
Anyway, it only slightly soured the night. Thankfully, I had the rest of "Run Lola Run" to make up for it...
Thursday, November 1, 2012
My Unfortunate Fraud Check
Every now and again I have an idea, and with that idea come thoughts of domain names. I like to think that I'm a positive guy, so I always hope against hope that I can use a 'word hack' to score a fun and useful domain.
Today I had one of those ideas, and that idea revolved around salaries. With my fingers crossed I gave the domain "salari.es" a shot in my browser address bar, but it didn't resolve to anything. That was definitely a start, but it didn't mean it was actually available. A quick search for ".es whois" and I was able to see if it was registered. To my amazement, it wasn't! I was super excited and ran (via my keyboard and browser) to my registrar of choice... which, it turns out, doesn't do .es domains. Ouch. Using the ".es whois" I was pushed to a particular site to register the domain, but for some reason I didn't feel comfortable using that site for registering the domain I wanted, so I kept looking.
I wondered if now was the time to try out another registrar I knew of: from everything I had heard, they were definitely trustworthy. I looked around, saw that they supported .es domains and quickly created an account.
I have to admit, I was super excited about this domain. It just seemed cool. I pulled out the first credit card I could found in my wallet, mash in the numbers and click 'Buy'.
Then I wait. And wait. And wait a little longer. This new registrar just kept telling me it was waiting for something... but I didn't know what. Until the phone rang. It was my credit card company in Australia calling to ensure that I had authorised payment to this new registrar, located in a country that is neither Australia (where my card "lives") nor Canada (where I live). Finally! I knew the hold up and so we were on our way again.
Or so I thought.
I'm not quite sure what the hold up was between then and there, to be honest, but suffice to say that it wasn't for at least a few more hours before things started to move again, and promptly failed. Turns out, someone else had registered it.
Now admittedly, I normally do these things a little differently. If I'm looking for a domain, I normally always use the search application in the registrar that I use. That way, if it's available, I buy it then and there. This time, however, they didn't support .es and I didn't feel like finding another registrar to sign up with, just to see if this seemingly awesome domain was already registered.
Don't read that the wrong way, though: I don't mean to imply that the initial site I was pushed to to register had anything to do with someone else registering the domain. It's just one of those things that make you question yourself.
Even if I hadn't done that, it still wouldn't have helped, as my credit card company was being fantastic, watching for and following up on possible online fraud. I can't fault them for that, either. If I had used the other one in my wallet, well, let's just say they don't seem to ever call me about these sorts of things :)
So I lost my first ever (cool) word hack domain name. At the end of the day, I got an account with a different registrar, and I don't expect I'll have any problems with the credit card company and them again. So that's a plus.
And to the new owner of "salari.es": I owe you a beer for being so lucky, and you probably owe one to my credit card company, too, for them being so particular. I hope you use it for something as awesome, ground-breaking, earth-shattering and paradigm-shifting as I was planning to...
Today I had one of those ideas, and that idea revolved around salaries. With my fingers crossed I gave the domain "salari.es" a shot in my browser address bar, but it didn't resolve to anything. That was definitely a start, but it didn't mean it was actually available. A quick search for ".es whois" and I was able to see if it was registered. To my amazement, it wasn't! I was super excited and ran (via my keyboard and browser) to my registrar of choice... which, it turns out, doesn't do .es domains. Ouch. Using the ".es whois" I was pushed to a particular site to register the domain, but for some reason I didn't feel comfortable using that site for registering the domain I wanted, so I kept looking.
I wondered if now was the time to try out another registrar I knew of: from everything I had heard, they were definitely trustworthy. I looked around, saw that they supported .es domains and quickly created an account.
I have to admit, I was super excited about this domain. It just seemed cool. I pulled out the first credit card I could found in my wallet, mash in the numbers and click 'Buy'.
Then I wait. And wait. And wait a little longer. This new registrar just kept telling me it was waiting for something... but I didn't know what. Until the phone rang. It was my credit card company in Australia calling to ensure that I had authorised payment to this new registrar, located in a country that is neither Australia (where my card "lives") nor Canada (where I live). Finally! I knew the hold up and so we were on our way again.
Or so I thought.
I'm not quite sure what the hold up was between then and there, to be honest, but suffice to say that it wasn't for at least a few more hours before things started to move again, and promptly failed. Turns out, someone else had registered it.
Now admittedly, I normally do these things a little differently. If I'm looking for a domain, I normally always use the search application in the registrar that I use. That way, if it's available, I buy it then and there. This time, however, they didn't support .es and I didn't feel like finding another registrar to sign up with, just to see if this seemingly awesome domain was already registered.
Don't read that the wrong way, though: I don't mean to imply that the initial site I was pushed to to register had anything to do with someone else registering the domain. It's just one of those things that make you question yourself.
Even if I hadn't done that, it still wouldn't have helped, as my credit card company was being fantastic, watching for and following up on possible online fraud. I can't fault them for that, either. If I had used the other one in my wallet, well, let's just say they don't seem to ever call me about these sorts of things :)
So I lost my first ever (cool) word hack domain name. At the end of the day, I got an account with a different registrar, and I don't expect I'll have any problems with the credit card company and them again. So that's a plus.
And to the new owner of "salari.es": I owe you a beer for being so lucky, and you probably owe one to my credit card company, too, for them being so particular. I hope you use it for something as awesome, ground-breaking, earth-shattering and paradigm-shifting as I was planning to...
Saturday, May 5, 2012
Farewell HealthcareIT.SE
So I know that I've been absent lately as far as posting is concerned... new jobs always tend to push me underground for a little while. I'll be back up to breathe again soon, but I daresay I might end up with less 'tech' and more 'life'. But we'll see.
I did want to take a quick moment though and just give a shout out to the now closed 'Healthcare IT Stack Exchange' site. This SE site got as far as public beta, but was closed yesterday due to essentially a lack of support.
Going in I knew that it would be a hard fought battle for that site to get traction, but I hoped against hope that it would. Yes, there are other support sites out there for healthcare IT providers. Maybe I haven't looked hard enough but from the few ones I've found, they seem to be either targeted at one particular aspect or product or their not free. Yes, there are other IT support sites out there and I use them regularly. In fact, the reason I knew about the healthcare IT Stack Exchange was because of the regular usage I make of Stack Overflow and other Stack Exchange sites. But don't be fooled: healthcare IT has a lot of nuances, acronyms and assumptions that mainstream IT does not. A normal IT support site cannot do what the dedicated healthcare IT SE site did.
I don't consider this to be a rant. It's just an observation. I find it hard to believe that there's not enough people to make a go of a site like that. I have no doubt, though, that getting enough people to know of it's existence is another problem altogether.
There are a lot of healthcare IT workers. Deep down, I wonder if the lack of support of this site is a reflection on the demographics more than anything. I get the feeling that in healthcare, you're almost always a massive conglomerate and yet I've always felt that Stack Overflow and it's SE offspring are more 'startup-y' than other support sites.
To me, at least, healthcare IT requires some startup shake-up. That's why I am where I am at the moment. It's why I'm also rooting for companies like Drchrono.
Anyway, back to the site: it was a great thing to see that site start up and it's a shame that it never materialised into what it could have become.
Let this just be my public thanks to Steve Wranovsky for proposing the site in the first place, and to all of the people that tried to make a go of it. I did my best to make it active, even if it was just an answer or two and an off topic question :)
Whatever the outcome, I know that there is a need for a site like this and hopefully it will come into existence at some point in the future, once we get enough startups focused on this well deserving industry.
I did want to take a quick moment though and just give a shout out to the now closed 'Healthcare IT Stack Exchange' site. This SE site got as far as public beta, but was closed yesterday due to essentially a lack of support.
Going in I knew that it would be a hard fought battle for that site to get traction, but I hoped against hope that it would. Yes, there are other support sites out there for healthcare IT providers. Maybe I haven't looked hard enough but from the few ones I've found, they seem to be either targeted at one particular aspect or product or their not free. Yes, there are other IT support sites out there and I use them regularly. In fact, the reason I knew about the healthcare IT Stack Exchange was because of the regular usage I make of Stack Overflow and other Stack Exchange sites. But don't be fooled: healthcare IT has a lot of nuances, acronyms and assumptions that mainstream IT does not. A normal IT support site cannot do what the dedicated healthcare IT SE site did.
I don't consider this to be a rant. It's just an observation. I find it hard to believe that there's not enough people to make a go of a site like that. I have no doubt, though, that getting enough people to know of it's existence is another problem altogether.
There are a lot of healthcare IT workers. Deep down, I wonder if the lack of support of this site is a reflection on the demographics more than anything. I get the feeling that in healthcare, you're almost always a massive conglomerate and yet I've always felt that Stack Overflow and it's SE offspring are more 'startup-y' than other support sites.
To me, at least, healthcare IT requires some startup shake-up. That's why I am where I am at the moment. It's why I'm also rooting for companies like Drchrono.
Anyway, back to the site: it was a great thing to see that site start up and it's a shame that it never materialised into what it could have become.
Let this just be my public thanks to Steve Wranovsky for proposing the site in the first place, and to all of the people that tried to make a go of it. I did my best to make it active, even if it was just an answer or two and an off topic question :)
Whatever the outcome, I know that there is a need for a site like this and hopefully it will come into existence at some point in the future, once we get enough startups focused on this well deserving industry.
Tuesday, March 20, 2012
Write Once Memory
I used to work with a colleague who I believe had what I liked to call 'write once memory'.
Basically, once they had been exposed to an idea or a solution, it was learned. Straight away. Which is great, really, in that you never have to go back over that stuff again. In theory. Unfortunately, there was also no way to retrain the concepts learned. Which, generally, is quite useful, too.
I remember one time when this person was working on a particular section of the web application that we were co-developing. Working on a particular page that, when reloaded, threw a web server error.
Straight away, they got up out of their seat and proceeded to restart a totally unrelated data service, because they had "seen that error before and it's the data service that causes it".
It wasn't until I walked out of the room and sent them the line from the web application's error log to show that, no, really, it was in the file that you were editing, that they would even *consider* the possibility that it was their mistake. This was not because of some egotistical mindset that refused to accept that they may have been wrong... it was simply that "I've seen x do y before and so when I see y, it must be x".
I know that sounds like an isolated incident, but time and again this person showed me that once they learned a connection, that was it for life.
It was odd and frustrating.
But it did teach me one thing: to this day, I try my hardest to ensure that I've given a problem the best assessment possible before going and making hair trigger attempts at resolving it. Along with this, I always try to work out the cause-and-effect relationship of what I type and what I ended up seeing.
Who knows... maybe I can use those assessments to realise how much I actually learned from that colleague of mine.
Basically, once they had been exposed to an idea or a solution, it was learned. Straight away. Which is great, really, in that you never have to go back over that stuff again. In theory. Unfortunately, there was also no way to retrain the concepts learned. Which, generally, is quite useful, too.
I remember one time when this person was working on a particular section of the web application that we were co-developing. Working on a particular page that, when reloaded, threw a web server error.
Straight away, they got up out of their seat and proceeded to restart a totally unrelated data service, because they had "seen that error before and it's the data service that causes it".
It wasn't until I walked out of the room and sent them the line from the web application's error log to show that, no, really, it was in the file that you were editing, that they would even *consider* the possibility that it was their mistake. This was not because of some egotistical mindset that refused to accept that they may have been wrong... it was simply that "I've seen x do y before and so when I see y, it must be x".
I know that sounds like an isolated incident, but time and again this person showed me that once they learned a connection, that was it for life.
It was odd and frustrating.
But it did teach me one thing: to this day, I try my hardest to ensure that I've given a problem the best assessment possible before going and making hair trigger attempts at resolving it. Along with this, I always try to work out the cause-and-effect relationship of what I type and what I ended up seeing.
Who knows... maybe I can use those assessments to realise how much I actually learned from that colleague of mine.
Tuesday, March 13, 2012
What You Sell and What They Buy
So this post may be rambling. I apologise in advance...
My dad was a pretty good business man. I, honestly, didn't care too much for 'being in business' as a kid but I did try and pay attention to him. When I was in high school, he started talking about a book called "Behind the Arches", which is a fairly hefty book about the McDonald's corporation, right from the start of it.
I learned lots of interesting tidbits from that book, but the basis of one story sticks in my head. Because this is all looking back a *long* time I really can't remember the specifics but basically it came down to this: when explaining the process of the business to some students, they were asked "so, what business is McDonald's in?". The answer that was given was "fast food".
That answer was wrong.
You see, while McDonald's might be selling burgers, it was said, it's actually in the business of real estate. This is to do with the fact that the McDonald's corporation tends to own the real estate that most of the franchises are built on. A good portion of the income of the McDonald's corporation is from drawing rent, basically. When the time comes (theoretically? :) ) that the franchise is no longer making money, all of a sudden McDonald's has land to sell. And it has a *lot* of land. In a *lot* of high traffic areas.
What you sell is not necessarily your business.
One other bizarre scene from my life has stuck in my head, and it basically talks about the flip side of this little statement. I was at a town hall meeting once (actually, again while I was in high school) and it was attended by a gentleman from the Rocky Mountain Institute. He was talking about renewable resources and recycling etc., looking at it from a "this is good for business" point of view.
He talked about how companies were beginning to realise that what people are buying is not the product, but what the product provided. From there, these companies were realising that leasing products was viable and finally, therefore, the more they can reuse product the better.
There were two examples that I can remember. Firstly, there was the carpet company that realised that people that outfitted office spaces *really* didn't want to buy carpet: they did, however, *really* want to buy the fact that floors weren't hard and cold. This led the company to actually lease carpets, which then made it all the more financially wise to be able to reuse at much of that carpet as possible. The outcome was carpet that could be fed into a wood chipper, for all intents and purposes, and to be able to use 100% of the result to remake new carpet.
The other example was a company that made lifts for office buildings. Again, they realised that people don't want to buy a lift. Instead, people are actually just looking for a way to get to other floors quickly and efficiently. This led to leasing the elevators themselves, which then gave the company a financial incentive to make the lifts as service-able and as reusable as possible.
So that became the reverse of the first story: people aren't buying things for any reason other than the fact that they need that thing to get a result they are actually looking for.
If a mass produced, safe and fast teleporter was built tomorrow, would anyone be actually buying lifts? Of course not. But that is not something to be scared of, even if you make lifts! Why? Because you're not making lifts... you just happen to need to right now to fulfill the requirements of the customer. Because while you may be selling lifts, you're in the business of getting people around buildings...
It's really odd that those two things stuck out to me, because I always just imagined that it was coincidence. The more I think about it, though, I really see that connection and I'm glad that I did pick up on it.
But what does that have to do with you? Well, assuming you're like me and in the IT industry, I think it distills down to something really simple: people aren't paying you to *produce* anything, they're *actually* paying you to get them a way to do something they couldn't before.
Do you really need to write up a whole framework of code to justify the price you put on a site? Not at all! They're not paying you to write code: they're paying you to get them a result.
Just the other day, I had a first time coder ask me "how long would it take me to write a web shop?". I delved a little deeper and the reason he wanted to know how long it would take was so that he could work out how much to charge! Obviously, outside of my current point, I convinced him not to write his own (like I say, this was really his first ever 'real project'), but I just suggested finding a product that already does e-commerce and basically theming it. Then when it came time to price it, he was amazed that he could charge that for such little work... until he realised that he wasn't being paid to write code: he was being paid to get results, and that result has a price on it, no matter how it is achieved.
So what's the moral? This one is especially aimed at the coders at the start of their career: the world doesn't need another framework, but there sure are a lot of people that will pay you to get themselves further ahead. You don't need to write it to justify the cost: you're being paid for information.
You might sell code, but information is your business.
My dad was a pretty good business man. I, honestly, didn't care too much for 'being in business' as a kid but I did try and pay attention to him. When I was in high school, he started talking about a book called "Behind the Arches", which is a fairly hefty book about the McDonald's corporation, right from the start of it.
I learned lots of interesting tidbits from that book, but the basis of one story sticks in my head. Because this is all looking back a *long* time I really can't remember the specifics but basically it came down to this: when explaining the process of the business to some students, they were asked "so, what business is McDonald's in?". The answer that was given was "fast food".
That answer was wrong.
You see, while McDonald's might be selling burgers, it was said, it's actually in the business of real estate. This is to do with the fact that the McDonald's corporation tends to own the real estate that most of the franchises are built on. A good portion of the income of the McDonald's corporation is from drawing rent, basically. When the time comes (theoretically? :) ) that the franchise is no longer making money, all of a sudden McDonald's has land to sell. And it has a *lot* of land. In a *lot* of high traffic areas.
What you sell is not necessarily your business.
One other bizarre scene from my life has stuck in my head, and it basically talks about the flip side of this little statement. I was at a town hall meeting once (actually, again while I was in high school) and it was attended by a gentleman from the Rocky Mountain Institute. He was talking about renewable resources and recycling etc., looking at it from a "this is good for business" point of view.
He talked about how companies were beginning to realise that what people are buying is not the product, but what the product provided. From there, these companies were realising that leasing products was viable and finally, therefore, the more they can reuse product the better.
There were two examples that I can remember. Firstly, there was the carpet company that realised that people that outfitted office spaces *really* didn't want to buy carpet: they did, however, *really* want to buy the fact that floors weren't hard and cold. This led the company to actually lease carpets, which then made it all the more financially wise to be able to reuse at much of that carpet as possible. The outcome was carpet that could be fed into a wood chipper, for all intents and purposes, and to be able to use 100% of the result to remake new carpet.
The other example was a company that made lifts for office buildings. Again, they realised that people don't want to buy a lift. Instead, people are actually just looking for a way to get to other floors quickly and efficiently. This led to leasing the elevators themselves, which then gave the company a financial incentive to make the lifts as service-able and as reusable as possible.
So that became the reverse of the first story: people aren't buying things for any reason other than the fact that they need that thing to get a result they are actually looking for.
If a mass produced, safe and fast teleporter was built tomorrow, would anyone be actually buying lifts? Of course not. But that is not something to be scared of, even if you make lifts! Why? Because you're not making lifts... you just happen to need to right now to fulfill the requirements of the customer. Because while you may be selling lifts, you're in the business of getting people around buildings...
It's really odd that those two things stuck out to me, because I always just imagined that it was coincidence. The more I think about it, though, I really see that connection and I'm glad that I did pick up on it.
But what does that have to do with you? Well, assuming you're like me and in the IT industry, I think it distills down to something really simple: people aren't paying you to *produce* anything, they're *actually* paying you to get them a way to do something they couldn't before.
Do you really need to write up a whole framework of code to justify the price you put on a site? Not at all! They're not paying you to write code: they're paying you to get them a result.
Just the other day, I had a first time coder ask me "how long would it take me to write a web shop?". I delved a little deeper and the reason he wanted to know how long it would take was so that he could work out how much to charge! Obviously, outside of my current point, I convinced him not to write his own (like I say, this was really his first ever 'real project'), but I just suggested finding a product that already does e-commerce and basically theming it. Then when it came time to price it, he was amazed that he could charge that for such little work... until he realised that he wasn't being paid to write code: he was being paid to get results, and that result has a price on it, no matter how it is achieved.
So what's the moral? This one is especially aimed at the coders at the start of their career: the world doesn't need another framework, but there sure are a lot of people that will pay you to get themselves further ahead. You don't need to write it to justify the cost: you're being paid for information.
You might sell code, but information is your business.
Monday, February 6, 2012
Setting Up Percona, Galera and SphinxSE
In the last project I worked on, I had a chance to play with what I would consider improvements to the typical MySQL install that most of us use. There were three aspects to what we were looking for:
As far as the MySQL fork is concerned: there are a number of projects that are based off of MySQL that have either improved or replaced the main InnoDB storage engine. These generally provide faster services than the traditional MySQL but some also add some extra features. For this, we wanted to try the Percona Server.
Multimaster replication in MySQL generally is a hassle to setup: not so much on the 'creation' side but on the redundancy side. It's pretty easy to set it up to run, but setting it up to fail 'correctly' generally involves other systems. We had heard of Galera and figured that this might be a good time to try it. Galera is basically a system that links in with the MySQL server/s, but at the same time talks amongst itself to ensure that all of the servers are in sync. If a new server comes up or an old server goes down, Galera is there talking, making sure that everything is still organised correctly.
Sphinx is basically a super fast full text searching service: it is much faster than, and more capable than, the traditional MySQL text searches. While Sphinx has a couple of different ways it can be interacted with, the Sphinx Storage Engine intrigued us. SphinxSE allows you to essentially tunnel through from a 'normal' MySQL table right through to Sphinx allowing you to do the Sphinx searching as 'just another' INNER JOIN in your query.
So with that in mind, we went searching for a solution that was already made :)
Unfortunately we weren't able to find anything complete... they were either missing the Galera component or the SphinxSE part.
So, for anyone interested in setting something like this up themselves, I've put together a quick little hacky guide to try. Hopefully everything makes as much sense as it can. If you read the instructions you'll see that we are using some custom (and possibly strange) directories for things like where the Sphinx data will go, as well as the MySQL/Percona data/configuration etc. You will also notice that there's a point where you download a patch file to apply to Sphinx: this is based on an older patch file for Percona that was aimed at an older version of Sphinx. It just makes sure that the SE will build.
Some final notes: I did all of this a while ago and so some things have changed since then. Galera has been updated since I wrote these instructions, as has Sphinx. I also just saw that Percona is expecting a beta release at the end of February 2012 of their 'Percona XtraDB Cluster' server which, from what I understand, is Percona with Galera built into it. So with that in mind, this is really just some notes on how to get things working if you're super keen on at least playing with this stuff. Oh yeah, and these notes are all based off of Ubuntu 11.10 Server. It assumes that you haven't already installed MySQL, as that gets to be a problem (as you're basically going to be trying to replace it). And with that in mind, here goes...
Be nice to yourself: become root...
Update and get some things you'll need...
Make your newly configured application:
Now, we enable Galera...
Edit
Enable the WSREP user so that it can read from and write to the databases:
Finally, run MySQL from the command line to get the CLI and run:
- Naturally we were looking for a system that had the reliability of MySQL, but at the same time we were itching to try some of the forks that have evolved from that codebase.
- We were hoping to get a better multimaster redundancy system setup than the traditional process.
- We were hoping to get SphinxSE involved.
As far as the MySQL fork is concerned: there are a number of projects that are based off of MySQL that have either improved or replaced the main InnoDB storage engine. These generally provide faster services than the traditional MySQL but some also add some extra features. For this, we wanted to try the Percona Server.
Multimaster replication in MySQL generally is a hassle to setup: not so much on the 'creation' side but on the redundancy side. It's pretty easy to set it up to run, but setting it up to fail 'correctly' generally involves other systems. We had heard of Galera and figured that this might be a good time to try it. Galera is basically a system that links in with the MySQL server/s, but at the same time talks amongst itself to ensure that all of the servers are in sync. If a new server comes up or an old server goes down, Galera is there talking, making sure that everything is still organised correctly.
Sphinx is basically a super fast full text searching service: it is much faster than, and more capable than, the traditional MySQL text searches. While Sphinx has a couple of different ways it can be interacted with, the Sphinx Storage Engine intrigued us. SphinxSE allows you to essentially tunnel through from a 'normal' MySQL table right through to Sphinx allowing you to do the Sphinx searching as 'just another' INNER JOIN in your query.
So with that in mind, we went searching for a solution that was already made :)
Unfortunately we weren't able to find anything complete... they were either missing the Galera component or the SphinxSE part.
So, for anyone interested in setting something like this up themselves, I've put together a quick little hacky guide to try. Hopefully everything makes as much sense as it can. If you read the instructions you'll see that we are using some custom (and possibly strange) directories for things like where the Sphinx data will go, as well as the MySQL/Percona data/configuration etc. You will also notice that there's a point where you download a patch file to apply to Sphinx: this is based on an older patch file for Percona that was aimed at an older version of Sphinx. It just makes sure that the SE will build.
Some final notes: I did all of this a while ago and so some things have changed since then. Galera has been updated since I wrote these instructions, as has Sphinx. I also just saw that Percona is expecting a beta release at the end of February 2012 of their 'Percona XtraDB Cluster' server which, from what I understand, is Percona with Galera built into it. So with that in mind, this is really just some notes on how to get things working if you're super keen on at least playing with this stuff. Oh yeah, and these notes are all based off of Ubuntu 11.10 Server. It assumes that you haven't already installed MySQL, as that gets to be a problem (as you're basically going to be trying to replace it). And with that in mind, here goes...
Be nice to yourself: become root...
sudo -sMake sure that for all of the main repositories (at least) you not only have the deb repos, but also the deb-src repos. This can be checked in
/etc/apt/sources.list
.Update and get some things you'll need...
apt-get update apt-get install dpkg-dev bzr cmake cmake-curses-gui chkconfig libssl0.9.8 apt-get build-dep mysql-serverCreate a MySQL user: this would normally be done by the MySQL installation.
groupadd mysql useradd -g mysql mysqlGet some directories ready.
mkdir /etc/mysql mkdir /etc/mysql/conf.d mkdir /var/lib/mysql mkdir /var/lib/mysql/data chown -R mysql:mysql /etc/mysql chown -R mysql:mysql /etc/mysql/conf.d chown -R mysql:mysql /var/lib/mysql mkdir /etc/sphinx mkdir /var/lib/sphinx mkdir /var/lib/sphinx/log mkdir /var/lib/sphinx/dataStart getting source...
cd /usr/local/src wget http://launchpad.net/galera/1.x/21.1.0/+download/galera-21.1.0-amd64.deb dpkg -i galera-21.1.0-amd64.deb bzr branch lp:~vadim-tk/percona-server/percona-5.5.15-galera wget http://www.sphinxsearch.com/files/sphinx-2.0.1-beta.tar.gz tar -zxvf sphinx-2.0.1-beta.tar.gz wget http://files.codefromaway.net/sphinx-2.0.1-beta-to-sphinx-2.0.1-beta-5.5.15.patch cd sphinx-2.0.1-beta patch -p1 < ../sphinx-2.0.1-beta-to-sphinx-2.0.1-beta-5.5.15.patch cp -R mysqlse ../percona-5.5.15-galera/storageBuild and install Percona
cd ../percona-5.5.15-galera ccmake .You will be given a UI at this point... press 'c' for configure, then 'e' to exit help then finally 't' to toggle to advanced mode. Now, scroll through the list and edit the following values (setting them to the values listed):
CMAKE_INSTALL_PREFIX /usr/bin/mysql INSTALL_MYSQLDATADIR /var/lib/mysql/data MYSQL_DATADIR /var/lib/mysql/data SYSCONFDIR /etc/mysql WITH_READLINE ON WITH_WSREP ONNow press 'c' for configure, 'e' to exit help and then 'g' to finally generate the configuration file.
Make your newly configured application:
make make test make installBuild and install Sphinx
cd /usr/local/src/sphinx-2.0.1-beta ./configure --prefix=/usr/bin/sphinx --sysconfdir=/etc/sphinx --libdir=/usr/libs --datarootdir=/var/lib/sphinx --with-mysql --with-mysql-includes=/usr/bin/mysql/include --with-mysql-libs=/usr/bin/mysql/lib make installPrepare some shared libraries. Create
/etc/ld.so.conf.d/mysql.conf
and then:
echo "/usr/bin/mysql/lib" > /etc/ld.so.conf.d/mysql.conf ldconfigStart getting Percona ready by copying configuration files as well as the init script for the service:
cd /usr/bin/mysql cp ./support-files/my-medium.cnf /etc/mysql/my.cnf cp ./support-files/wsrep.cnf /etc/mysql/conf.d/wsrep.cnf cp ./support-files/mysql.server /etc/init.dEdit the
/etc/mysql/conf.d/wsrep.cnf
file and set the following values:wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_cluster_address="gcomm://" wsrep_sst_auth=wsrep_sst:some_random_passwordNOTE: the wsrep_cluster_address line is initially commented out! you have to remove the commenting hash AS WELL as make the gcomm change... Also note that the 'some_random_password' will get used towards the end of this whole process, so remember it!
Now, we enable Galera...
Edit
/etc/mysql/my.cnf
: add the following lines to the end of the file:binlog_format=row !includedir /etc/mysql/conf.dEdit
/etc/bash.bashrc
and add the following lines:PATH=$PATH:/usr/bin/mysql/bin:/usr/bin/sphinx/bin export PATHReload bashrc so that the MySQL binaries and Sphinx binaries are now in your path:
source /etc/bash.bashrc source ~/.bashrcPrepare the MySQL database and finally start the service!
./scripts/mysql_install_db --basedir=/usr/bin/mysql --user=mysql --datadir=/var/lib/mysql/data /etc/init.d/mysql.server startUse the following two lines to get MySQL to load on startup:
chkconfig --list mysql.server chkconfig mysql.server on --level 2,3,4,5Restart the server now, just to be sure :)
Enable the WSREP user so that it can read from and write to the databases:
mysql -u root -e "SET wsrep_on=OFF; DELETE FROM mysql.user WHERE user='';" mysql -u root -e "SET wsrep_on=OFF; GRANT ALL ON *.* TO wsrep_sst@'%' IDENTIFIED BY 'some_random_password';"'some_random_password' is what we remembered back up towards the middle of this process.
Finally, run MySQL from the command line to get the CLI and run:
INSTALL PLUGIN sphinx SONAME 'ha_sphinx.so';You should now have Percona up and running with Galera replication and SphinxSE on the side. Have fun... I hope it helps *someone*!
Sunday, January 29, 2012
Simple Cachebusting for JS
Browsers caching your assets is a good thing. It not only reduces the bandwidth requirements of your site, but it also speeds up loading on the client side every time they visit.
But what happens when you have some changes that you need pushed everywhere to have the site still working? The big thing I'm thinking of here is Javascript. You change the HTML format of a page and all of a sudden your jQuery fails, or your AJAX responses have been tweaked and now your 'onSuccess' functions that work with that data need to be changed... and if they're not, then things don't work.
So what do you do? Of course, there's always the option to stop caching: but that will increase server load, used bandwidth and slow down the loading on the client side. Even if you do disable caching, you're still at the mercy of the client... maybe the browser decides to ignore your cache control requests (or possibly the proxies between the client and the server ignore the requests).
So in trying to work this out, I realised that by adding a random GET parameter will help resolve this issue. Every browser that I have tested against treats the (partial) URL of "random.js?12345" as entirely different from "random.js?12346" as far as caching is concerned, and with good reason. The GET parameter is used for exactly that: to "get" a given resource. It's understandable that the browser would treat these two things as different.
I could use a random number every time to force the cache busting, but that basically sends me back to losing any benefit of browser caching. So the question becomes, what do I use for the GET parameter? Initially I considered using a version number. Every time I changed the JS file, I would change the URL that the generated HTML would use... it would go from "random.js?1.2.3" to "random.js?1.2.4" and so on. In thinking about that implementation, though, I realised that my laziness and forgetfulness would be a deal breaker here as there is no way I would be able to remember to update the URL in (every?) file that references the JS file.
What I ended up doing was actually pretty simple. Every time I include a JS file in the HTML generated by a PHP script, I first quickly find the last modified time of the file and I then append that modified timestamp to the URL.
For my random.js example above, I would do something like the following in my index.php file:
$stats = stat(dirname(__FILE__) . '/random.js');
echo '<script src="random.js?' . $stats['mtime'] . '"/>';
Really simple, perfectly functional and highly cross platform, which is something I always strive to do. If you're not using PHP, then the concept is still the same... you would just need to translate it to your favourite programming language.
Now, if you have a lot of resources that you want to include, maybe you would write a function to try and remove the redundancy, but generally when I do this I only have one or two files to process anyway.
Naturally, the better alternative is to use a framework that does this all for you anyway. A lot of those frameworks do a lot more for you than just the cache busting, so of course they're worth the effort, but if you're not in a situation where you can choose this route, then hopefully the above will help!
But what happens when you have some changes that you need pushed everywhere to have the site still working? The big thing I'm thinking of here is Javascript. You change the HTML format of a page and all of a sudden your jQuery fails, or your AJAX responses have been tweaked and now your 'onSuccess' functions that work with that data need to be changed... and if they're not, then things don't work.
So what do you do? Of course, there's always the option to stop caching: but that will increase server load, used bandwidth and slow down the loading on the client side. Even if you do disable caching, you're still at the mercy of the client... maybe the browser decides to ignore your cache control requests (or possibly the proxies between the client and the server ignore the requests).
So in trying to work this out, I realised that by adding a random GET parameter will help resolve this issue. Every browser that I have tested against treats the (partial) URL of "random.js?12345" as entirely different from "random.js?12346" as far as caching is concerned, and with good reason. The GET parameter is used for exactly that: to "get" a given resource. It's understandable that the browser would treat these two things as different.
I could use a random number every time to force the cache busting, but that basically sends me back to losing any benefit of browser caching. So the question becomes, what do I use for the GET parameter? Initially I considered using a version number. Every time I changed the JS file, I would change the URL that the generated HTML would use... it would go from "random.js?1.2.3" to "random.js?1.2.4" and so on. In thinking about that implementation, though, I realised that my laziness and forgetfulness would be a deal breaker here as there is no way I would be able to remember to update the URL in (every?) file that references the JS file.
What I ended up doing was actually pretty simple. Every time I include a JS file in the HTML generated by a PHP script, I first quickly find the last modified time of the file and I then append that modified timestamp to the URL.
For my random.js example above, I would do something like the following in my index.php file:
$stats = stat(dirname(__FILE__) . '/random.js');
echo '<script src="random.js?' . $stats['mtime'] . '"/>';
Really simple, perfectly functional and highly cross platform, which is something I always strive to do. If you're not using PHP, then the concept is still the same... you would just need to translate it to your favourite programming language.
Now, if you have a lot of resources that you want to include, maybe you would write a function to try and remove the redundancy, but generally when I do this I only have one or two files to process anyway.
Naturally, the better alternative is to use a framework that does this all for you anyway. A lot of those frameworks do a lot more for you than just the cache busting, so of course they're worth the effort, but if you're not in a situation where you can choose this route, then hopefully the above will help!
Monday, January 16, 2012
Will the PHP Community Ever Drop the "Silly n00b" Presumption?
A long, long time ago, I learned my first scripting language: Perl. Admittedly, I wasn't doing any database interactions and to be honest the code wasn't entirely complex (it was a basic user management system with access to different data sets depending on account information).
I wouldn't say that I asked a lot of questions, but I definitely *read* a lot of them and not once can I remember anyone basically answering a question by just pointing out an insecurity in the original posters' question.
Cut now to my current web language of choice: PHP. It seems as though more often than not, one of the first answers to a question online is pointing out the security issues with the question, and a lot of the time, the original question isn't even being answered!
Time and time again, "SQL injection!" or some such stuff is the answer, even when the question had nothing to do with it. It's repetitive and to be completely honest, a little boring now. I almost feel as though Person A got hit with one of these replies once and so now they feel as though they have to pass on the 'love' to Person B.
I don't know what it is that's causing this... there's just too many variables. Is it just a sign of the times? I don't know: I don't spend much time in other language support fields. Is it just because I wasn't aware of these types of replies before? Again, I don't know. Maybe it's just me being a cranky old man looking back on the good old days. Maybe, too, Perl went through these growing pains and I was just fortunate enough to be on the other side of them.
I will go out on a limb, though, and guess it's a combination of two things: 1) the gamification of questions and answers; and 2) the increase in people teaching themselves to code.
Not that either of these things are a problem in and of themselves. Having said that, sites like stackoverflow.com have really made it worthwhile to try and get in and get some upvotes and I hate to say it, but the PHP section of that site is looking less and less usable to me every day. At the same time, more and more people are learning to code in a 'non-professional' environment. Again, this isn't a bad thing, however I get the feeling that these basic security issues seem to have been missed the first time over when people teach themselves so to them, it's amazingly impressive to be able to find these issues with someone elses code, to the point where the answer to the actual question doesn't matter as much as this.
Maybe I'm just tired, but I just feel as though at some point, the PHP community as a whole needs to get over this whole thing about "commenting on security problems is my number one concern, even if I can't answer the original question"... it just isn't helping anyone.
Don't get me wrong: a good, correct answer is fantastic, even if there's an aside of "BTW, SQL injection!" but please, that's not always the answer!
I wouldn't say that I asked a lot of questions, but I definitely *read* a lot of them and not once can I remember anyone basically answering a question by just pointing out an insecurity in the original posters' question.
Cut now to my current web language of choice: PHP. It seems as though more often than not, one of the first answers to a question online is pointing out the security issues with the question, and a lot of the time, the original question isn't even being answered!
Time and time again, "SQL injection!" or some such stuff is the answer, even when the question had nothing to do with it. It's repetitive and to be completely honest, a little boring now. I almost feel as though Person A got hit with one of these replies once and so now they feel as though they have to pass on the 'love' to Person B.
I don't know what it is that's causing this... there's just too many variables. Is it just a sign of the times? I don't know: I don't spend much time in other language support fields. Is it just because I wasn't aware of these types of replies before? Again, I don't know. Maybe it's just me being a cranky old man looking back on the good old days. Maybe, too, Perl went through these growing pains and I was just fortunate enough to be on the other side of them.
I will go out on a limb, though, and guess it's a combination of two things: 1) the gamification of questions and answers; and 2) the increase in people teaching themselves to code.
Not that either of these things are a problem in and of themselves. Having said that, sites like stackoverflow.com have really made it worthwhile to try and get in and get some upvotes and I hate to say it, but the PHP section of that site is looking less and less usable to me every day. At the same time, more and more people are learning to code in a 'non-professional' environment. Again, this isn't a bad thing, however I get the feeling that these basic security issues seem to have been missed the first time over when people teach themselves so to them, it's amazingly impressive to be able to find these issues with someone elses code, to the point where the answer to the actual question doesn't matter as much as this.
Maybe I'm just tired, but I just feel as though at some point, the PHP community as a whole needs to get over this whole thing about "commenting on security problems is my number one concern, even if I can't answer the original question"... it just isn't helping anyone.
Don't get me wrong: a good, correct answer is fantastic, even if there's an aside of "BTW, SQL injection!" but please, that's not always the answer!
Sunday, January 8, 2012
Would You Flattr My Wishlist?
The other day I was reading a blog post somewhere that answered enough of my questions that I saved a significant amount of time. It was the first time that I was driven to really try and find a way to get the poster some sign of appreciation.
The closest thing I could find, though, was an Amazon wishlist. That was fine, however the cheapest thing on that list was around $60. Don't get me wrong, I appreciated the information, but it was a big mental leap from never having given anything to a blog poster to giving $60.
It was at that point that I realised that there was no way to give a 'partial payment' to a wishlist item. From what I could find, anyway.
This is my request, to anyone that cares: please make it possible for me to just give a random amount of money to an item on a wishlist. That way I can give something that really matters to the wishlist owner, and it's a little more 'personal' than sending a gift card / credit / whatever it is that I would have had to have done otherwise.
Then I started thinking about flattr.com: maybe they could do something similar, by integrating their system so that as a person that can be 'flattred', the money goes towards items on my wishlist or something similar.
To me, at least, the outcome of this is way cooler than getting a bit of money in some electronic account... it would be awesome to just every now and again get a random item out of my wishlist as a total surprise!
Does anyone know of anyone working on this?
The closest thing I could find, though, was an Amazon wishlist. That was fine, however the cheapest thing on that list was around $60. Don't get me wrong, I appreciated the information, but it was a big mental leap from never having given anything to a blog poster to giving $60.
It was at that point that I realised that there was no way to give a 'partial payment' to a wishlist item. From what I could find, anyway.
This is my request, to anyone that cares: please make it possible for me to just give a random amount of money to an item on a wishlist. That way I can give something that really matters to the wishlist owner, and it's a little more 'personal' than sending a gift card / credit / whatever it is that I would have had to have done otherwise.
Then I started thinking about flattr.com: maybe they could do something similar, by integrating their system so that as a person that can be 'flattred', the money goes towards items on my wishlist or something similar.
To me, at least, the outcome of this is way cooler than getting a bit of money in some electronic account... it would be awesome to just every now and again get a random item out of my wishlist as a total surprise!
Does anyone know of anyone working on this?
Subscribe to:
Posts (Atom)