Ersin Acar » MySQL

Using Fulltext Indexes in MySQL – Part 2, Boolean searches

Ersin Acar — Fri, 26 Mar 2010 18:53:06 +0000

Part 1 of this article looked at the fulltext index, and how to search on it using an ordinary MATCH() AGAINST(). Even more powerful, (although only available on the newer MySQL version 4), is the ability to do a boolean search. Part 2 of this article examines the possibilities.

You’ll use the same table you used in Part 1. The full list of records is:

mysql> SELECT * FROM fulltext_sample;
+---------------------------+
| copy                      |
+---------------------------+
| It appears good from here |
| The here and the past     |
| Why are we here           |
| An all-out alert          |
| All you need is love      |
| A good alert              |
+---------------------------+

To perform a boolean search, the IN BOOLEAN MODE modifier is used. The following query demonstrates the new syntax.

mysql> SELECT * FROM fulltext_sample
WHERE MATCH (copy) AGAINST ('love' IN BOOLEAN MODE);
+----------------------+
| copy                 |
+----------------------+
| All you need is love |
+----------------------+

An important difference between ordinary MATCH() AGAINST() queries and IN BOOLEAN MODE searches is that the latter does not apply the 50% limit (so the word can appear in more than half the rows).

There are a number of special operators you’ll need to know in order to make the most of the boolean search. If no symbol is specified, the appearance of this word causes the relevance of the row to be higher, similar to an ordinary MATCH() AGAINST().

Boolean Search Operators

+	The word is mandatory in all rows returned.
-	The word cannot appear in any row returned.
<	The word that follows has a lower relevance than other words, although rows containing it will still match
>	The word that follows has a higher relevance than other words.
()	Used to group words into subexpressions.
~	The word following contributes negatively to the relevance of the row (which is different to the ‘-’ operator, which specifically excludes the word, or the ‘<’ operator, which still causes the word to contribute positively to the relevance of the row.
*	The wildcard, indicating zero or more characters. It can only appear at the end of a word.
“	Anything enclosed in the double quotes is taken as a whole (so you can match phrases, for example).

Examples

Let’s see some of this action. The first example returns all rows containing the word ‘here’, but not the word ‘past’.

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE);
+---------------------------+
| copy                      |
+---------------------------+
| It appears good from here |
| Why are we here           |
+---------------------------+

Compare the difference between the next two examples:

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('here past' IN BOOLEAN MODE);
+---------------------------+
| copy                      |
+---------------------------+
| It appears good from here |
| The here and the past     |
| Why are we here           |
+---------------------------+

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('here +past'IN BOOLEAN MODE);
+-----------------------+
| copy                  |
+-----------------------+
| The here and the past |
+-----------------------+

The first works in a similar way to an ordinary MATCH() AGAINST(), albeit without the 50% threshold. The word ‘past’ is mandatory in the second search (by default a word is optional), so the other two records are not returned.

The next example demonstrates a common mistake.

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('+are here' IN BOOLEAN MODE);
+---------------------------+
| copy                      |
+---------------------------+
| It appears good from here |
| The here and the past     |
| Why are we here           |
+---------------------------+

The results may appear surprising compared the previous example, but since ‘are’ contains three or less letters, it is excluded for purposes of the search and is not mandatory.

The next two examples demonstrate a powerful enhancement for searching purposes:

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('aler' IN BOOLEAN MODE);
Empty set (0.01 sec)

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('aler*' IN BOOLEAN MODE);
+------------------+
| copy             |
+------------------+
| An all-out alert |
| A good alert     |
+------------------+

By default only whole words are matched, unless the '*' operator is used.

The '<' and '>' symbols are less commonly used, but they allow a great degree of control for relevance. In the following examples, we return the relevance indicator to demonstrate the difference between the queries.
mysql> SELECT copy, MATCH(copy)
AGAINST ('appears good alert' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('appears good alert' IN BOOLEAN MODE);
+---------------------------+------+
| copy                      | m    |
+---------------------------+------+
| It appears good from here |    2 |
| An all-out alert          |    1 |
| A good alert              |    2 |
+---------------------------+------+

mysql> SELECT copy, MATCH(copy)
AGAINST ('appears 
The '<' operator decreases the relevance of the word 'good', in this case by approximately 0.33.
mysql> SELECT copy, MATCH(copy)
AGAINST ('appears good >alert' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('appears good >alert' IN BOOLEAN MODE);
+---------------------------+------+
| copy                      | m    |
+---------------------------+------+
| It appears good from here |    2 |
| An all-out alert          |  1.5 |
| A good alert              |  2.5 |
+---------------------------+------+
The '>' operator increases the relevance of the word 'alert', in this case by 0.5.
The parentheses group words into a subexpression. In the following example, the '+' symbol applies to the group of words, so that at least one of 'appears' or 'past' must appear.
mysql> SELECT copy, MATCH(copy)
AGAINST ('+(appears past)' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('(appears past)' IN BOOLEAN MODE);
+---------------------------+------+
| copy                      | m    |
+---------------------------+------+
| It appears good from here |    1 |
| The here and the past     |    1 |
+---------------------------+------+
You can also apply the operators to words in the subexpression, as follows:
mysql> SELECT copy, MATCH(copy)
AGAINST ('+(>appears appears 
The '~' operator contributes negatively to the relevance, but does not bar the word from appearing, as the '-' operator does. It also does not do the same as the '<' operator, which is to make a reduced, yet still positive, difference to the relevance. These subtle difference allows for powerfully tuned searches, but can cause confusion. Look at the difference between the following. First, the word 'here' is compulsory, and the word 'past' optional:
mysql> SELECT copy, MATCH(copy)
AGAINST ('+here past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here past' IN BOOLEAN MODE);
+---------------------------+-----------------+
| copy                      | m               |
+---------------------------+-----------------+
| It appears good from here |               1 |
| The here and the past     | 1.3333333730698 |
| Why are we here           |               1 |
+---------------------------+-----------------+
Next, a reduced relevance for the word 'past'. The same results are returned, but the record with both words ('The here and the past') has less weighting, though still more than the other records.
mysql> SELECT copy, MATCH(copy)
AGAINST ('+here 
Take note of the relevance in the next example:
mysql> SELECT copy, MATCH(copy)
AGAINST ('+here ~past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here ~past' IN BOOLEAN MODE);
+---------------------------+------------------+
| copy                      | m                |
+---------------------------+------------------+
| It appears good from here |                1 |
| The here and the past     | 0.83333331346512 |
| Why are we here           |                1 |
+---------------------------+------------------+
The important difference is that the relevance is negatively affected, and is now lower than the other two records. This means if you asked for only '~past', you'd get nothing back, as the relevance would be lower than 0 (and therefore not returned)
mysql> SELECT copy, MATCH(copy)
AGAINST ('~past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('~past' IN BOOLEAN MODE);
Empty set (0.00 sec)
Finally, '-' simply removes the second record from the result set, as we've seen before.
mysql> SELECT copy, MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE);
+---------------------------+------+
| copy                      | m    |
+---------------------------+------+
| It appears good from here |    1 |
| Why are we here           |    1 |
+---------------------------+------+
The final operator, the double quotes ("), is useful to group phrases together. So, if you were interested in returning 'good from here' but none of those words if they did not appear in the phrase, you'd use:
mysql> SELECT copy, MATCH(copy)
AGAINST ('"good from here"' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('"good from here"' IN BOOLEAN MODE);
+---------------------------+------+
| copy                      | m    |
+---------------------------+------+
| It appears good from here |    1 |
+---------------------------+------+
For many applications, boolean searches are one of MySQL 4's most useful features. With careful use, you can save yourself substantial of development time, and add many useful enhancements. Good luck!

Mysql Dump and Restore commands

Ersin Acar — Thu, 21 May 2009 17:50:55 +0000

if you want to backup and restore your big MySQL databases you need better solution than phpmyadmin

So here we go;

To dump all databases to a file:

$ mysqldump -u dbuser -pdbpass --all-databases > all-databases.sql

To dump a single database to a file:

$ mysqldump -u dbuser -pdbpass somedb > somedb.sql

Dump InnoDB in a single transaction:

$ mysqldump -u dbuser -pdbpass --single-transaction somedb > somedb.sql

To restore:

$ mysql -u dbuser -pdbpass somedb < somedb.sql

Change/Fix Wrong MySQL Encoding

Ersin Acar — Fri, 24 Apr 2009 09:37:00 +0000

Sometimes, especially when moving data from one server to another, you might find that you have your MySQL database incorrectly encoded. Problem with this first show itself if you have the database and encoded in one charset set to display your website in another. If this is the case then you will find strange characters appearing in your text, especially when using punctuation marks. If you are Unwilling or unable to change the character encoding on the site then you need to change how the data is encoded in the database.

The most common sort of thing you might want to do is change from iso-8859-1 (or windows-1252) to UTF-8. This can be done in one of two ways.

The first way is to simply alter the table so that the column contains a different charset.

ALTER TABLE table MODIFY col1 VARCHAR(50) CHARACTER SET 'utf8';

However, if your database has already been set up and your data has already been inserted in the wrong format then you can also update the data in the column using the CONVERT command. The following snippet turns our latin1 data into uncoded binary data and then into utf8.

UPDATE table SET col1=CONVERT(CONVERT(CONVERT(col1 USING 'latin1') USING BINARY) USING 'utf8');

You should also make sure that the connection to the database is done through a specific character set. This is done by using the

SET NAMES command and the SET CHARACTER SET.
 
SET NAMES 'charset_name'
SET CHARACTER SET 'charset_name';

These two commands basically set some values in your MySQL database, for more information on what is set look at the Connection Character Sets and Collations page on the MySQL website. This ensures that the data we get back from the database is also in the correct charset.

For a full list of the different character sets available in MySQL just run the command:

SHOW CHARACTER SET;

This will display a table with the columns Charset, Description, Default collation and Maxlen. Each charset is associated with a collation. A collation is a set of rules for comparing characters in a charset, so it is important that you get this right if you want the database to work. The full list of collations can be viewed using the following command:

SHOW COLLATION;

You can even use a LIKE statement to refine the collation data into the information you are looking for.

SHOW COLLATION WHERE Charset LIKE '%utf%'

Deleting dublicated rows in MySQL

Ersin Acar — Thu, 09 Apr 2009 08:44:09 +0000

To find duplicate values you need to use the MySQL COUNT() function and then pick out all of the counts that are greater than one.

SELECT VALUE,COUNT(VALUE) AS COUNT
FROM test
GROUP BY VALUE
HAVING (COUNT(VALUE) > 1)
ORDER BY COUNT DESC;

Conversely you can also select the rows that only have a single entry.

SELECT VALUE,COUNT(VALUE) AS COUNT
FROM test
GROUP BY VALUE
HAVING (COUNT(VALUE) =; 1)
ORDER BY COUNT DESC;

However, it is very nice to pick out the duplicate entries in a table, but you might still need to do something with them. Here is a query to delete any duplicate rows from a table. It does a simple self join and deletes the row value with the lowest ID.

DELETE bad_rows.*
FROM tests AS good_rows
INNER JOIN tests AS bad_rows ON bad_rows.NUMBER = good_rows.NUMBER
AND bad_rows.id > good_rows.id;

More information on this deletion query and other methods of deleting duplicates can be found at Xaprb.com.

Fix Wrong Character Encoding In MySQL

Ersin Acar — Wed, 08 Apr 2009 22:31:04 +0000

Sometimes, especially when moving data from one server to another, you might find that you have encoded your MySQL database incorrectly. This problem with first show itself if you have the database encoded in one charset and your website set to display in another. If this is the case then you will find strange characters appearing in your text, especially when using punctuation marks. If you are unable or unwilling to change the character encoding on the site then you need to change how the data is encoded in the database.

The most common sort of thing you might want to do is change from iso-8859-1 (or windows-1252) to UTF-8. This can be done in one of two ways.

The first way is to simply alter the table so that the column contains a different charset.

ALTER TABLE TABLE MODIFY col1 VARCHAR(50) CHARACTER SET 'utf8';

UPDATE TABLE SET col1=CONVERT(CONVERT(CONVERT(col1 USING 'latin1') USING BINARY) USING 'utf8');

You should also make sure that the connection to the database is done through a specific character set. This is done by using the SET NAMES command and the SET CHARACTER SET.

SET NAMES 'charset_name'
SET CHARACTER SET 'charset_name';

For a full list of the different character sets available in MySQL just run the command:

SHOW CHARACTER SET;

SHOW COLLATION;

You can even use a LIKE statement to refine the collation data into the information you are looking for.

SHOW COLLATION WHERE Charset LIKE '%utf%'