I have set up a full text search to handle multiple columns searching
for chinese
But the result of the search isn't really what i have expected.
I have setup the catalog to have a chiense word break, and the columns
in the tables are all nvachar
when i do something like
select * from dbo.Table_1 where contains(*, '"<chinese
character>"',language 1082)
the search result is really inconsistent, especially with single
characters.I have also checked that these characters are not in the
noise filter file...
the search result is better when the input is more than a single
characters, but still, somtimes it will not return any result at all.
so, I try to use the "like" statement instead of "contains" to do the
search with the same inputs, and 100% of the time, it returns the
correct result.
does anyone have any experience about things like that? coz I guess
this is a more spcific issue with language. Is there any place that
you guys know of, can offer me some help?
thank you in advance.
On Feb 11, 5:33Xam, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out characters to try figure out which characters go together and which ones should be treated as a single "word". It even pulls out radicals.
> However, try this on the thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on 某人為了商X或個人使用.
> Then try 某人為了商X或個人使 (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
icic
oh Thank you, so there is no way to improve that ?
thank you
|||On Feb 11, 5:33Xam, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out characters to try figure out which characters go together and which ones should be treated as a single "word". It even pulls out radicals.
> However, try this on the thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on 某人為了商X或個人使用.
> Then try 某人為了商X或個人使 (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
I hit the send button too quickly for the last post...
coz I tried out a couple of single character that my search couldn't
handle in http://forums.asp.net, and it acutally can return results
there...
i am just wondering if there is any other settings i can try to
improve that ?
thank you
|||You may have better success using the Traditional word breaker.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
<admin.onQhk@.gmail.com> wrote in message
news:1171225568.325092.171990@.h3g2000cwc.googlegro ups.com...
On Feb 11, 5:33 am, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out
> characters to try figure out which characters go together and which ones
> should be treated as a single "word". It even pulls out radicals.
> However, try this on the
> thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on ???.
> Then try ?? (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication
> book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL
> FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in
> messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
icic
oh Thank you, so there is no way to improve that ?
thank you
|||They may be doing a like.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
<admin.onQhk@.gmail.com> wrote in message
news:1171226670.382397.154660@.k78g2000cwa.googlegr oups.com...
On Feb 11, 5:33 am, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out
> characters to try figure out which characters go together and which ones
> should be treated as a single "word". It even pulls out radicals.
> However, try this on the
> thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on ???.
> Then try ?? (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication
> book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL
> FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in
> messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
I hit the send button too quickly for the last post...
coz I tried out a couple of single character that my search couldn't
handle in http://forums.asp.net, and it acutally can return results
there...
i am just wondering if there is any other settings i can try to
improve that ?
thank you
2012年3月27日星期二
full text search with language other than english(ex chinese, japanese)
I have set up a full text search to handle multiple columns searching for chinese
But the result of the search isn't really what i have expected.
I have setup the catalog to have a chiense word break, and the columns in the tables are all nvachar
when i do something like
select * from dbo.Table_1 where contains(*, '"<chinese character>"',language 1082)
the search result is really inconsistent, especially with single characters.I have also checked that these characters are not in the noise filter file...
the search result is better when the input is more than a single characters, but still, somtimes it will not return any result at all.
so, I try to use the "like" statement instead of "contains" to do the search with the same inputs, and 100% of the time, it returns the correct result.
does anyone have any experience about things like that? coz I guess this is a more spcific issue with language. Is there any place that you guys know of, can offer me some help?
thank you in advance.
I think you need to use the correct Chinese collation instead of just Nvarchar because Nvarchar just tells SQL Server you are not using ASCII but Chinese and Japanese alphabet are more than two thousand characters while Latin is just 26. There are six Chinese and I think three Japanese collations defined in SQL Server, try the link below to choose your collation. I also think you need to make sure your Microsoft search catalog is populated and check the noise word file. Run a search for noise words and Microsoft search catalog in SQL Server BOL(books online). Hope this helps.
http://msdn2.microsoft.com/en-us/library/ms180175.aspx
|||Thank you so much
But after trying lots... of different collation settings, it still yields the same results...
Is there any more think that I can try ?
thank you
|||
Out of cuorcuriosity, I try to search a single chinese character which my search couldn't handle in this asp.net forums. This forum's search acutally returns result ...
so there must be something wrong that I am doing coz I really doubt that they will do anything special in this forum to handle chinese characters specifically......
just wondering if there is anything else that i could try ??
thank you in advance.....
2012年3月26日星期一
Full text search on Chinese or Chinese/English mix?
I want to use SQL 2005 FT to search on web page I crawled from web. The
page can be Chinese, English or Chinese/English(Chinese article with English
phrase in it).
First question is that what language word breaker I should choose. Does
Chinese word breaker make its English content hard to search?.
Second question, Should I store text in different language in difference
catalog so that I can choose the specific word breaker for the FTS? but how
to determine what language a web page is using. Most of Chinese and English
web page uses utf-8 charset which make it indistinguishable for my program
to determine which language it is using. Shouldn't SQL server figure out
what word breaker to use automattically by examining the bytes of utf-8
encoding of the text?
Third, what encoding I should use when I insert the content of web page into
the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
matter?
Your inputs are greatly appreciated.
You have to use the ms.locale metatag for this to work, store your documents
in the image or varbinary data type, and then query using the Language
keyword. The language type you assign to the column is irrelevant as the
langauge tags in the document type dominate. Here is an example
CREATE TABLE blob
(pk INT not null IDENTITY(1,1) CONSTRAINT primarykey PRIMARY KEY,
blob VARBINARY(MAX),
blobtype VARCHAR(10))
GO
CREATE FULLTEXT INDEX ON blob
(blob TYPE COLUMN blobtype LANGUAGE 1033) --note the LCID is for American
English
KEY INDEX PrimaryKey ON catalog_name
GO
--note that these html documents we are pushing in are tagged with French
language metatags.
INSERT INTO blob (blob,blobtype)
VALUES(CONVERT(VARBINARY(256),'<HTML><HEAD><META name="ms.locale"
CONTENT="FR"></HEAD><BODY>mang</BODY></HTML>'),'.htm')
INSERT INTO blob (blob,blobtype)
VALUES(CONVERT(VARBINARY(256),'<HTML><HEAD><META name="ms.locale"
CONTENT="FR"></HEAD><BODY>manger</BODY></HTML>'),'.htm')
GO
Querying for all stemmed forms of the French verb manger (to eat).
SELECT * FROM blob WHERE CONTAINS(*, 'formsof(inflectional,manger), language
1036)
--two rows returned.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Xin Chen" <xchen@.xtremework.com> wrote in message
news:uReDrmxJFHA.3184@.TK2MSFTNGP09.phx.gbl...
> I want to use SQL 2005 FT to search on web page I crawled from web. The
> page can be Chinese, English or Chinese/English(Chinese article with
English
> phrase in it).
> First question is that what language word breaker I should choose. Does
> Chinese word breaker make its English content hard to search?.
> Second question, Should I store text in different language in difference
> catalog so that I can choose the specific word breaker for the FTS? but
how
> to determine what language a web page is using. Most of Chinese and
English
> web page uses utf-8 charset which make it indistinguishable for my program
> to determine which language it is using. Shouldn't SQL server figure out
> what word breaker to use automattically by examining the bytes of utf-8
> encoding of the text?
> Third, what encoding I should use when I insert the content of web page
into
> the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
> matter?
> Your inputs are greatly appreciated.
>
|||Maybe I didn't answer your question to well.
1) It doesn't matter what word breaker you select as for varbinary or image
data type columns where the document's contains language tags the iFilter
understands (HTML docs tagged with the ms.locale metatag, or Word and other
Office docs) the embedded language tag will control the word breaker used.
2) You don't have to if you are using the Image or varbinary data type
columns. For other data type columns you will.
3) utf-8 should work.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Xin Chen" <xchen@.xtremework.com> wrote in message
news:uReDrmxJFHA.3184@.TK2MSFTNGP09.phx.gbl...
> I want to use SQL 2005 FT to search on web page I crawled from web. The
> page can be Chinese, English or Chinese/English(Chinese article with
English
> phrase in it).
> First question is that what language word breaker I should choose. Does
> Chinese word breaker make its English content hard to search?.
> Second question, Should I store text in different language in difference
> catalog so that I can choose the specific word breaker for the FTS? but
how
> to determine what language a web page is using. Most of Chinese and
English
> web page uses utf-8 charset which make it indistinguishable for my program
> to determine which language it is using. Shouldn't SQL server figure out
> what word breaker to use automattically by examining the bytes of utf-8
> encoding of the text?
> Third, what encoding I should use when I insert the content of web page
into
> the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
> matter?
> Your inputs are greatly appreciated.
>
page can be Chinese, English or Chinese/English(Chinese article with English
phrase in it).
First question is that what language word breaker I should choose. Does
Chinese word breaker make its English content hard to search?.
Second question, Should I store text in different language in difference
catalog so that I can choose the specific word breaker for the FTS? but how
to determine what language a web page is using. Most of Chinese and English
web page uses utf-8 charset which make it indistinguishable for my program
to determine which language it is using. Shouldn't SQL server figure out
what word breaker to use automattically by examining the bytes of utf-8
encoding of the text?
Third, what encoding I should use when I insert the content of web page into
the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
matter?
Your inputs are greatly appreciated.
You have to use the ms.locale metatag for this to work, store your documents
in the image or varbinary data type, and then query using the Language
keyword. The language type you assign to the column is irrelevant as the
langauge tags in the document type dominate. Here is an example
CREATE TABLE blob
(pk INT not null IDENTITY(1,1) CONSTRAINT primarykey PRIMARY KEY,
blob VARBINARY(MAX),
blobtype VARCHAR(10))
GO
CREATE FULLTEXT INDEX ON blob
(blob TYPE COLUMN blobtype LANGUAGE 1033) --note the LCID is for American
English
KEY INDEX PrimaryKey ON catalog_name
GO
--note that these html documents we are pushing in are tagged with French
language metatags.
INSERT INTO blob (blob,blobtype)
VALUES(CONVERT(VARBINARY(256),'<HTML><HEAD><META name="ms.locale"
CONTENT="FR"></HEAD><BODY>mang</BODY></HTML>'),'.htm')
INSERT INTO blob (blob,blobtype)
VALUES(CONVERT(VARBINARY(256),'<HTML><HEAD><META name="ms.locale"
CONTENT="FR"></HEAD><BODY>manger</BODY></HTML>'),'.htm')
GO
Querying for all stemmed forms of the French verb manger (to eat).
SELECT * FROM blob WHERE CONTAINS(*, 'formsof(inflectional,manger), language
1036)
--two rows returned.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Xin Chen" <xchen@.xtremework.com> wrote in message
news:uReDrmxJFHA.3184@.TK2MSFTNGP09.phx.gbl...
> I want to use SQL 2005 FT to search on web page I crawled from web. The
> page can be Chinese, English or Chinese/English(Chinese article with
English
> phrase in it).
> First question is that what language word breaker I should choose. Does
> Chinese word breaker make its English content hard to search?.
> Second question, Should I store text in different language in difference
> catalog so that I can choose the specific word breaker for the FTS? but
how
> to determine what language a web page is using. Most of Chinese and
English
> web page uses utf-8 charset which make it indistinguishable for my program
> to determine which language it is using. Shouldn't SQL server figure out
> what word breaker to use automattically by examining the bytes of utf-8
> encoding of the text?
> Third, what encoding I should use when I insert the content of web page
into
> the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
> matter?
> Your inputs are greatly appreciated.
>
|||Maybe I didn't answer your question to well.
1) It doesn't matter what word breaker you select as for varbinary or image
data type columns where the document's contains language tags the iFilter
understands (HTML docs tagged with the ms.locale metatag, or Word and other
Office docs) the embedded language tag will control the word breaker used.
2) You don't have to if you are using the Image or varbinary data type
columns. For other data type columns you will.
3) utf-8 should work.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Xin Chen" <xchen@.xtremework.com> wrote in message
news:uReDrmxJFHA.3184@.TK2MSFTNGP09.phx.gbl...
> I want to use SQL 2005 FT to search on web page I crawled from web. The
> page can be Chinese, English or Chinese/English(Chinese article with
English
> phrase in it).
> First question is that what language word breaker I should choose. Does
> Chinese word breaker make its English content hard to search?.
> Second question, Should I store text in different language in difference
> catalog so that I can choose the specific word breaker for the FTS? but
how
> to determine what language a web page is using. Most of Chinese and
English
> web page uses utf-8 charset which make it indistinguishable for my program
> to determine which language it is using. Shouldn't SQL server figure out
> what word breaker to use automattically by examining the bytes of utf-8
> encoding of the text?
> Third, what encoding I should use when I insert the content of web page
into
> the full text database? use utf-8, or gb2312(chinese) or Unicode? Does it
> matter?
> Your inputs are greatly appreciated.
>
2012年3月22日星期四
Full text search for Arabic and English
Hi all,
I have a database which i use for Arabic and English text, i want to know if i can use the full text search for this database. The default language is Latin but i can recreate the db using Arabic BIN as default.
Thanks in advance
Haidar,
While SQL Server (7.0 and 2000 - Which version are you using?) both fully support the Arabic language, Full-Text Search only supports a "subset" of SQL Server supported languages and Arabic is not one of those languages. However, you can use the Neutral "Language for Word Breaker" when you setup the FT Catalog via the FT Indexing Wizard under SQL Server 2000. If you're using SQL Server 7.0,
you will have to set the entire server to the "Neutral" language setting via sp_configure. The supported subset of languages supported by SQL Server for
FTS is documented in the Books Online (BOL)....
In addition to the BOL documentation, there is now on MSDN - "Arabic Language Support in Microsoft SQL Server 2000" at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsql2k/html/sql_arabicsupport.asp that might also be helpful to you.
Finally, you might also want to download and test a Microsoft developed Arabic wordbreaker and you can download the Microsoft Arabic Word-Breaker (Arabic Search Engine) at
http://www.microsoft.com/middleeast/arabicdev/beta/search/ as this might be a solution to your problem!
Thanks,
John
--
SQL Full Text Search Blog
http://spaces.msn.com/members/jtkane/
2012年3月21日星期三
Full text Problems
Hi all, my english is very bad, so i try to explain...
when i put the string
and Contains (PG.GenericTitleFullTxt,'"* Dias *" AND "* que *"AND "* Abalaram *" AND "* Mundo * "')
in my query, the results show up.... but is only i put the " * O * " in the search string and not shows ..something linke
and Contains (PG.GenericTitleFullTxt,'"* Dias *" AND "* que *"AND "* Abalaram *" AND "* O * " AND * Mundo * "')
All the conditions for the "and" exists....in my catalog i have the string
Dias que Abalaram O Mundo
many times..bu if i put the "O" in the search ...not shows nothing...
I droped the catalog, indexes..rebuild..etc...
Anyone can help me ?
Thanks..
First of all, you should leave the blank before and after the words, as the word breaker has already broken the words in the database to small pieces which can be recognized. if the words you are searching for are all single words you should also leave the * signs due to the same reasons mentioned above.Jens K. Suessmeyer
http://www.sqlserver2005.de
full text problem
I have created a catalog from a table that contains only english with <html> tags. When i run a query against it i get
Server: Msg 7619, Level 16, State 1, Line 1
Langauge database/cache file could not be found.
I have read that setting full text to use the neutral word breaker may solve this problem, question is "How do i set it to use the neutral word breaker"
ThanksThe following articles may be helpful:
Article q246701 (http://support.microsoft.com/default.aspx?scid=kb;EN-US;q246701)
Article q271818 (http://support.microsoft.com/default.aspx?scid=kb;EN-US;q271818)
Server: Msg 7619, Level 16, State 1, Line 1
Langauge database/cache file could not be found.
I have read that setting full text to use the neutral word breaker may solve this problem, question is "How do i set it to use the neutral word breaker"
ThanksThe following articles may be helpful:
Article q246701 (http://support.microsoft.com/default.aspx?scid=kb;EN-US;q246701)
Article q271818 (http://support.microsoft.com/default.aspx?scid=kb;EN-US;q271818)
2012年3月19日星期一
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibiza
I'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting I love japanese ^_^
|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
please help!
thanks
ibiza
|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
someone please help!
thanks
ibiza
|||Does text indexing support non-latin characters?
|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all? ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibiza
I'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting
I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
please help!
thanks
ibiza
|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
someone please help!
thanks
ibiza
|||Does text indexing support non-latin characters?
|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all? ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibiza
This is probably due to the way the Japanese characters are broken by the
word breakers at index time and stored in the full text index. For example
Japanese consists of 5 different character sets. Japanese "words" are
largely syllables, so when you search on a "word" what it matches to are a
variety of sub tokens/syllables.
What you need to do is have someone who is fluent in Japanese verify that
your search application is actually finding what you are looking for. My
knowledge of Japanese is limited to a few phrases.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"ibiza" <lambertb@.gmail.com> wrote in message
news:1143825609.907248.76110@.t31g2000cwb.googlegro ups.com...
> Hi all,
> I am quite experimented with SQL Server, but not that much with full
> text indexing. After some successful attempts with english fields, I've
> decided to try it with Japanese characters. I don't know why, but it
> seems to have a strange behaviour.
>
> As in this screenshot
> (http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
> function does not seem to return only fields with an exact word match
> of the given "word" (query), but also strange results which does not
> even correspond to the query. Can anybody help me with that one?
>
> Thanks!
>
> ibiza
>
|||Hi,
thanks a lot for your reply
Well, even if I am not "fluent" in Japanese, I can still read japanese
"hiragana" characters, that is, what my query is about.
Each symbol can be translated as a syllable, so =E3=81=8A=E3=82=82=E3=81=97=
=E3=82=8D (the
query) would be "omoshiro", "o-mo-shi-ro".
The three first rows are translated as "omoni", "omo" and "omoi". So
I'm wondering why these results are showing, because they only partly
contains the query word (the =E3=81=8A=E3=82=82 ("omo") part)
As far as I know Japanese and full text indexing, the query should only
return results where the full and comple quey is found (=E3=81=8A=E3=82=82=
=E3=81=97=E3=82=8D),
as I did a CONTAINS search with a whole word.
Can you help me a little more with that information?
thanks again
ibiza
|||This could be by design. For example in German a word like wanderlust is
broken as wardern, lust and wanderlist. So if you were to do a contains
query for wanderlust you would get hits to any of these words. It looks like
it is doing something similar.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"ibiza" <lambertb@.gmail.com> wrote in message
news:1143922571.663177.238920@.i39g2000cwa.googlegr oups.com...
Hi,
thanks a lot for your reply
Well, even if I am not "fluent" in Japanese, I can still read japanese
"hiragana" characters, that is, what my query is about.
Each symbol can be translated as a syllable, so ? (the
query) would be "omoshiro", "o-mo-shi-ro".
The three first rows are translated as "omoni", "omo" and "omoi". So
I'm wondering why these results are showing, because they only partly
contains the query word (the ? ("omo") part)
As far as I know Japanese and full text indexing, the query should only
return results where the full and comple quey is found (?),
as I did a CONTAINS search with a whole word.
Can you help me a little more with that information?
thanks again
ibiza
|||Hi there,
thanks again for your reply.
I did some other tests with english and they all seemed successful
until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all? ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
thanks for your help!
ibiza
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibiza
This is probably due to the way the Japanese characters are broken by the
word breakers at index time and stored in the full text index. For example
Japanese consists of 5 different character sets. Japanese "words" are
largely syllables, so when you search on a "word" what it matches to are a
variety of sub tokens/syllables.
What you need to do is have someone who is fluent in Japanese verify that
your search application is actually finding what you are looking for. My
knowledge of Japanese is limited to a few phrases.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"ibiza" <lambertb@.gmail.com> wrote in message
news:1143825609.907248.76110@.t31g2000cwb.googlegro ups.com...
> Hi all,
> I am quite experimented with SQL Server, but not that much with full
> text indexing. After some successful attempts with english fields, I've
> decided to try it with Japanese characters. I don't know why, but it
> seems to have a strange behaviour.
>
> As in this screenshot
> (http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
> function does not seem to return only fields with an exact word match
> of the given "word" (query), but also strange results which does not
> even correspond to the query. Can anybody help me with that one?
>
> Thanks!
>
> ibiza
>
|||Hi,
thanks a lot for your reply
Well, even if I am not "fluent" in Japanese, I can still read japanese
"hiragana" characters, that is, what my query is about.
Each symbol can be translated as a syllable, so =E3=81=8A=E3=82=82=E3=81=97=
=E3=82=8D (the
query) would be "omoshiro", "o-mo-shi-ro".
The three first rows are translated as "omoni", "omo" and "omoi". So
I'm wondering why these results are showing, because they only partly
contains the query word (the =E3=81=8A=E3=82=82 ("omo") part)
As far as I know Japanese and full text indexing, the query should only
return results where the full and comple quey is found (=E3=81=8A=E3=82=82=
=E3=81=97=E3=82=8D),
as I did a CONTAINS search with a whole word.
Can you help me a little more with that information?
thanks again
ibiza
|||This could be by design. For example in German a word like wanderlust is
broken as wardern, lust and wanderlist. So if you were to do a contains
query for wanderlust you would get hits to any of these words. It looks like
it is doing something similar.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"ibiza" <lambertb@.gmail.com> wrote in message
news:1143922571.663177.238920@.i39g2000cwa.googlegr oups.com...
Hi,
thanks a lot for your reply
Well, even if I am not "fluent" in Japanese, I can still read japanese
"hiragana" characters, that is, what my query is about.
Each symbol can be translated as a syllable, so ? (the
query) would be "omoshiro", "o-mo-shi-ro".
The three first rows are translated as "omoni", "omo" and "omoi". So
I'm wondering why these results are showing, because they only partly
contains the query word (the ? ("omo") part)
As far as I know Japanese and full text indexing, the query should only
return results where the full and comple quey is found (?),
as I did a CONTAINS search with a whole word.
Can you help me a little more with that information?
thanks again
ibiza
|||Hi there,
thanks again for your reply.
I did some other tests with english and they all seemed successful
until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all? ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
thanks for your help!
ibiza
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibizaI'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
please help!
thanks
ibiza|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
someone please help!
thanks
ibiza|||Does text indexing support non-latin characters?|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks!
ibizaI'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting
I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
please help!
thanks
ibiza|||well, if you can help me with that problem, I'd be glad to consider it
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=
nce I
can find...
someone please help!
thanks
ibiza|||Does text indexing support non-latin characters?|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks! :)
ibizaI'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting :) I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=nce I
can find...
please help!
thanks :)
ibiza|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=nce I
can find...
someone please help!
thanks :)
ibiza|||Does text indexing support non-latin characters?|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks! :)
ibizaI'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting :) I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=nce I
can find...
please help!
thanks :)
ibiza|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "=E3=81=8A=E3=82=82", but that's the only resembla=nce I
can find...
someone please help!
thanks :)
ibiza|||Does text indexing support non-latin characters?|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks! :)
ibizaHi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks! :)
ibizaHi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all' ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
Full text indexing with Japanese characters problem
Hi all,
I am quite experimented with SQL Server, but not that much with full
text indexing. After some successful attempts with english fields, I've
decided to try it with Japanese characters. I don't know why, but it
seems to have a strange behaviour.
As in this screenshot
(http://img65.imageshack.us/img65/980/jap3xt.gif), the CONTAINS
function does not seem to return only fields with an exact word match
of the given "word" (query), but also strange results which does not
even correspond to the query. Can anybody help me with that one?
Thanks! :)
ibizaI'm not sure about your problem, but could I have a copy of your DB?
That looks amazingly interesting :) I love japanese ^_^|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "おも", but that's the only resemblance I
can find...
please help!
thanks :)
ibiza|||well, if you can help me with that problem, I'd be glad to consider it
:)
(be warned though, it's more than 500MB in size...)
But seriously, I really don't understand why is the full-text indexing
service getting me those "almost random" rows where the dic_kana column
does not contain an exact match...yes, both the query and most of the
returned rows stard with "おも", but that's the only resemblance I
can find...
someone please help!
thanks :)
ibiza|||Does text indexing support non-latin characters?|||Hi again,
I did some other tests with english this time, and they all seemed
successful...until this one :
http://img155.imageshack.us/img155/2001/untitled1tx.gif
Why is the CONTAINS returning no row at all?? ;_;
This makes me doubt about the validity of all the previous succesful
queries I made...Could there be missing words, if some queries like the
one with 'now' isn't returning any at all?
What's the problem with full text indexing? :S
And yes I did repopulate all the indexes before running this query...
help would be greatly appreciated!
thanks
ibiza
订阅：
博文 (Atom)