full text search with language other than english(ex chinese, japanese)

I have set up a full text search to handle multiple columns searching
for chinese
But the result of the search isn't really what i have expected.
I have setup the catalog to have a chiense word break, and the columns
in the tables are all nvachar
when i do something like
select * from dbo.Table_1 where contains(*, '"<chinese
character>"',language 1082)
the search result is really inconsistent, especially with single
characters.I have also checked that these characters are not in the
noise filter file...
the search result is better when the input is more than a single
characters, but still, somtimes it will not return any result at all.
so, I try to use the "like" statement instead of "contains" to do the
search with the same inputs, and 100% of the time, it returns the
correct result.
does anyone have any experience about things like that? coz I guess
this is a more spcific issue with language. Is there any place that
you guys know of, can offer me some help?
thank you in advance.
On Feb 11, 5:33Xam, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out characters to try figure out which characters go together and which ones should be treated as a single "word". It even pulls out radicals.
> However, try this on the thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on 某人為了商X或個人使用.
> Then try 某人為了商X或個人使 (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
icic
oh Thank you, so there is no way to improve that ?
thank you
|||On Feb 11, 5:33Xam, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out characters to try figure out which characters go together and which ones should be treated as a single "word". It even pulls out radicals.
> However, try this on the thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on 某人為了商X或個人使用.
> Then try 某人為了商X或個人使 (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
I hit the send button too quickly for the last post...
coz I tried out a couple of single character that my search couldn't
handle in http://forums.asp.net, and it acutally can return results
there...
i am just wondering if there is any other settings i can try to
improve that ?
thank you
|||You may have better success using the Traditional word breaker.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
<admin.onQhk@.gmail.com> wrote in message
news:1171225568.325092.171990@.h3g2000cwc.googlegro ups.com...
On Feb 11, 5:33 am, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out
> characters to try figure out which characters go together and which ones
> should be treated as a single "word". It even pulls out radicals.
> However, try this on the
> thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on ???.
> Then try ?? (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication
> book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL
> FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in
> messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
icic
oh Thank you, so there is no way to improve that ?
thank you
|||They may be doing a like.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
<admin.onQhk@.gmail.com> wrote in message
news:1171226670.382397.154660@.k78g2000cwa.googlegr oups.com...
On Feb 11, 5:33 am, "Hilary Cotter" <hilary.cot...@.gmail.com> wrote:
> The Chinese word breaker does multiple passes on characters pulling out
> characters to try figure out which characters go together and which ones
> should be treated as a single "word". It even pulls out radicals.
> However, try this on the
> thishttp://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> searching on ???.
> Then try ?? (note I have removed the last character)
> http://search.microsoft.com/results.aspx?q=%E6%9F%90%E4%BA%BA%E7%82%B...
> Nothing found.
> So while it works well in general, there are some inconsistencies.
> --
> Hilary Cotter
> Looking for a SQL Server replication
> book?http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL
> FTShttp://www.indexserverfaq.com
>
> <admin.on...@.gmail.com> wrote in
> messagenews:1171171251.557135.10240@.h3g2000cwc.goo glegroups.com...
>
>
>
>
>
> - Show quoted text -
I hit the send button too quickly for the last post...
coz I tried out a couple of single character that my search couldn't
handle in http://forums.asp.net, and it acutally can return results
there...
i am just wondering if there is any other settings i can try to
improve that ?
thank you

full text search with language other than english(ex chinese, japanese)

I have set up a full text search to handle multiple columns searching for chinese

But the result of the search isn't really what i have expected.

I have setup the catalog to have a chiense word break, and the columns in the tables are all nvachar

when i do something like

select * from dbo.Table_1 where contains(*, '"<chinese character>"',language 1082)

the search result is really inconsistent, especially with single characters.I have also checked that these characters are not in the noise filter file...

the search result is better when the input is more than a single characters, but still, somtimes it will not return any result at all.

so, I try to use the "like" statement instead of "contains" to do the search with the same inputs, and 100% of the time, it returns the correct result.

does anyone have any experience about things like that? coz I guess this is a more spcific issue with language. Is there any place that you guys know of, can offer me some help?

thank you in advance.

I think you need to use the correct Chinese collation instead of just Nvarchar because Nvarchar just tells SQL Server you are not using ASCII but Chinese and Japanese alphabet are more than two thousand characters while Latin is just 26. There are six Chinese and I think three Japanese collations defined in SQL Server, try the link below to choose your collation. I also think you need to make sure your Microsoft search catalog is populated and check the noise word file. Run a search for noise words and Microsoft search catalog in SQL Server BOL(books online). Hope this helps.

http://msdn2.microsoft.com/en-us/library/ms180175.aspx

|||

Thank you so much

But after trying lots... of different collation settings, it still yields the same results...
Is there any more think that I can try ?

thank you

|||

Out of cuorcuriosity, I try to search a single chinese character which my search couldn't handle in this asp.net forums. This forum's search acutally returns result ...
so there must be something wrong that I am doing coz I really doubt that they will do anything special in this forum to handle chinese characters specifically......

just wondering if there is anything else that i could try ??

thank you in advance.....

Full Text Search on Patterns in SQL Server 2005

Can the full text search in SQL Server 2005 handle cases like a wildcard in
the middle of a word?
Will
Hi Will,
Since SQL Server 2005 is not released yet, I could recommand you posting
here
http://forums.microsoft.com/msdn/def...orumGroupID=19
Thank you for your patience and cooperation. If you have any questions or
concerns, don't hesitate to let me know. We are always here to be of
assistance!
Sincerely yours,
Michael Cheng
Microsoft Online Partner Support
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
================================================== ===
This posting is provided "AS IS" with no warranties, and confers no rights.

Full Text Search on Patterns in SQL Server 2005

Can the full text search in SQL Server 2005 handle cases like a wildcard in
the middle of a word?
WillHi Will,
Since SQL Server 2005 is not released yet, I could recommand you posting
here
http://forums.microsoft.com/msdn/de...ForumGroupID=19
Thank you for your patience and cooperation. If you have any questions or
concerns, don't hesitate to let me know. We are always here to be of
assistance!
Sincerely yours,
Michael Cheng
Microsoft Online Partner Support
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
========================================
=============
This posting is provided "AS IS" with no warranties, and confers no rights.

Full Text Search on Patterns in SQL Server 2005

Can the full text search in SQL Server 2005 handle cases like a wildcard in
the middle of a word?
--
WillHi Will,
Since SQL Server 2005 is not released yet, I could recommand you posting
here
http://forums.microsoft.com/msdn/default.aspx?ForumGroupID=19
Thank you for your patience and cooperation. If you have any questions or
concerns, don't hesitate to let me know. We are always here to be of
assistance!
Sincerely yours,
Michael Cheng
Microsoft Online Partner Support
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
=====================================================This posting is provided "AS IS" with no warranties, and confers no rights.

FTS capabilities + questions

Hello,
I have some questions about Full Text Search and how to handle a
scenario that I have. The database that I am going to hit up against
has 80 tables and over 250 fields that will be searched for FTS. This
database has 600 + tables in it so 80 is needed to make the search
meaningfull for the user.
What is the best method to use FTS against so many tables?
Could I build a massive query that dumps the results into a single
table for searching?
Can I use the 80 tables and use some kind of dynamic query to do my
joins so that it scales?
I know this problem isnt unique to our company but just wondering how
some of the gurus have solved this problem?
Thanks in advance.
That really depends on the queries you have. Ideally you will have a single
table which you full text index. The queries themselves against the
full-text indexes are processed separately from the queries against the
tables, so it is somewhat asynchronous.
Adding this asynchronous nature to 80 tables means that you do not have a
scalable solution.
HTH
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
<tmueller@.teewebco.com> wrote in message
news:1163098034.044676.194200@.h48g2000cwc.googlegr oups.com...
> Hello,
> I have some questions about Full Text Search and how to handle a
> scenario that I have. The database that I am going to hit up against
> has 80 tables and over 250 fields that will be searched for FTS. This
> database has 600 + tables in it so 80 is needed to make the search
> meaningfull for the user.
> What is the best method to use FTS against so many tables?
> Could I build a massive query that dumps the results into a single
> table for searching?
> Can I use the 80 tables and use some kind of dynamic query to do my
> joins so that it scales?
>
> I know this problem isnt unique to our company but just wondering how
> some of the gurus have solved this problem?
> Thanks in advance.
>
|||Thanks for the fast response.
How about doing the queries up front and stuffing the results into a
single field that is searchable?
Is this the preferred method when encountering many tables like me?
Hilary Cotter wrote:[vbcol=seagreen]
> That really depends on the queries you have. Ideally you will have a single
> table which you full text index. The queries themselves against the
> full-text indexes are processed separately from the queries against the
> tables, so it is somewhat asynchronous.
> Adding this asynchronous nature to 80 tables means that you do not have a
> scalable solution.
> HTH
> --
> Hilary Cotter
> Director of Text Mining and Database Strategy
> RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
> This posting is my own and doesn't necessarily represent RelevantNoise's
> positions, strategies or opinions.
> Looking for a SQL Server replication book?
> http://www.nwsu.com/0974973602.html
> Looking for a FAQ on Indexing Services/SQL FTS
> http://www.indexserverfaq.com
>
> <tmueller@.teewebco.com> wrote in message
> news:1163098034.044676.194200@.h48g2000cwc.googlegr oups.com...
|||Sure, this is an option, but then you would have to full-text index this on
the fly. One other option comes to mind, and that is if you are using SQL
2005, you might be able to full-text index an index view which represents
your underlying tables.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"teewebco" <tmueller@.teewebco.com> wrote in message
news:1163101288.921588.258730@.f16g2000cwb.googlegr oups.com...
> Thanks for the fast response.
> How about doing the queries up front and stuffing the results into a
> single field that is searchable?
> Is this the preferred method when encountering many tables like me?
>
>
> Hilary Cotter wrote:
>
|||Ill take a look at that. Thanks!