Finding links between thoughts

G

Guest

Guest
Hi!

So I am using a digital voice recorder to record anything I see important, ranging from business to private, from rants to new business ideas.

Every finalized idea is one wma-file.

I wrote a program to sort the wma-files into folders. From time to time I listen to the wma-files, convert them to text(manually) and insert them into a mindmap with mindmanager, which I sort hierarchically by area and type in turn.

This works very well, no idea is being lost, when I am out of ideas for a special topic, I listen to what I said and can get started again.

What could a search system look like that finds links between thoughts(written in the mindmap and in the wma files) or in general gives me good search results even when the keyword I searched for is not present but a synonym of it or related topic(for instance flower should output entries containing orchid aswell, even if they contain orchid but not the very keyword flower).

I prefer something ready-made but small adjustments to a given system are fine aswell.

How would you approach this task?
 

clango

Registered
Tanz der Nacht;104072 said:
Hi!

I prefer something ready-made but small adjustments to a given system are fine aswell.

How would you approach this task?

The first idea coming out is to convert in next action what you catch. Then if you use an electronic task manager ( Toodledo and Ultimate To Do list) you should add in the note a lot of choosen key words or you could use tags. A common tag should be the link between the thoughts you are looking for
 

cwoodgold

Registered
Tanz der Nacht;104087 said:
I don't understand what you mean.

I think what clango means is, that when you convert it to text, look at what you're typing and put in some extra words; for example, if you write something with the word "orchid" in it, put the word "flower" in there too as an extra word, so that later when you search for "flower" you'll find it.

I do something like that. I have a text file on the computer listing the paper files in my filing cabinet. When I put a paper file in, I add a line to the text file on the computer to name the file and tell me where it is, and I add some more words and phrases to the line, guessing things that I might search for when looking for that file.
For example, for a "passport" file, I might type in keywords such as "id", "identification", and "renewal date", so that later I can find it when searching for those phrases.
 
G

Guest

Guest
The goal is to find links I have never thought about. You need to think different.
 

cwoodgold

Registered
Here's a website which might be a step in the right direction: http://www.wordslike.net/ WordsLike.net is a free service that allows you to find words and phrases that are similar or related to each other.

Your question reminded me of the following somewhat-related ideas:

Although it may not solve your problem, this website came to mind:
http://eigentaste.berkeley.edu/user/index.php
People can add jokes, and rate the jokes; and the idea is that after you've rated a few jokes, the website will supply you with jokes that tend to be liked by people who like the same jokes that you like. This goes a step further than the rating systems on some websites that merely show you things that tend to be liked by people in general, or things that are liked by people who like one particular thing. In other words, it tries to show you stuff you'll like to see. They may not have collected a very large number of jokes and ratings yet. I'd like to see websites like this for other categories of stuff, such as links to news articles, movies, blog posts, etc.

Hypothes.is is another interesting website. They have a plan to facilitate crowd-sourced annotation of all websites, with ratings so that you can look at what have
been chosen as the most interesting or appropriate comments first.

I like the book "Fluid Concepts and Creative Analogies" by Douglas Hofstadter.

As for your question: I'm not aware of software to do that. I think people at places
like Google might be working on stuff like that, or at least I imagine they
ought to, or at least I'd like them to.

Software could work like this: Take a very large corpus such as a collection of a large
number of web pages. Rate words or phrases as related if they appear more often near each other (e.g. within a couple of paragraphs) than randomly. Given two or more words or phrases, rate other words or phrases as interesting if they're related to more than one of the other words or phrases. A score for how interesting could be calculated. Print out the most interesting ones.

For example, a Google search for "orchid" returns results in which the word "flower" appears within the first page or two of snippets.

For some things, the human brain still works better than any computer program.
They've written programs to play chess, although not in the same creative, imaginative, purposeful way that humans play it. They've been making progress
on things like ability to read handwriting, but the progress is slow. Someone
speculated that in a couple of decades when computers may have about as much
computing power as the human brain then they may do a lot of the stuff humans do;
but maybe not, since it's also necessary to write the computer programs to organize
all that computing power. Maybe programs will write other programs, but I'm not
sure if we're anywhere near doing anything much useful with that. (OK, I have
a program just for fun that writes a copy of itself, and another program that usefully writes the repetitive stuff that comes at the beginning of
other programs I want to write, but those programs aren't being what I call creative.)

Here's an example of something I think the human mind does better than a
computer program. Suppose one major step in solving a cryptic crossword clue
is to find a word which, perhaps with a bit of a stretch of the imagination and
perhaps in two different senses, can be seen as fitting each of two different
definitions which are supplied. (See "Double definition" in the Wikipedia
article on cryptic crosswords; example: "not seeing window covering" has
the answer "blind".) First of all, the human mind will probably do better at
recognizing in the first place that this clue is likely of the double-definition type.
Secondly, while a program might find the two linkages to the word "blind",
it might well completely miss it because it says "not seeing" rather than
"unable to see", etc.

To solve this type of clue, I might not use a step-by-step method, which might
miss the answer just as a computer program might. Instead, I go into a different
frame of mind, sort-of relaxing the mind, focus on both definitions at once in
an intuitive sort of way, and try to imagine what the answer might be like.
I might also go off and work on other clues or leave it to the next day so my
subconscious mind can work on it. I think that by solving cryptic crosswords
I've learned to access a state of mind I wasn't really aware of before.

Martin Ternouth (see two emails by Ternouth here http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=00008c) talks about the human mind
making connections and says "impossible to do on computer".

So, my approach (which need not be the same as your approach) would be
to look over the material regularly as Ternouth suggests, perhaps varying it
somehow -- shuffling the order, rewriting it in different words, etc. --
and wait for ideas to come bubbling up out of my subconscious.
Writing down your dreams can help, too. And solving cryptic crosswords.

My page on "Optimism and Creativity" talks about techniques, many by
Edward de Bono, for thinking more creatively: http://web.ncf.ca/an588/create.html

I hope you find a computer program that does what you're talking about.
Let me know if you do, because I'm interested in that kind of thing.
 
G

Guest

Guest
For the following I want to note that I am german. So if you recommend any service with a given database, it has to work with german phrases(aswell).

As for your question: I'm not aware of software to do that. I think people at places
like Google might be working on stuff like that, or at least I imagine they
ought to, or at least I'd like them to.
Nice.

Google brought the prediction api to us, which is a kind of uclassify. But you cannot test the pa without billing and uclassify, well, might work. But for privacy reasons I prefer a local version. After all it is quite private what I am collecting there obviously.

Software could work like this: Take a very large corpus such as a collection of a large
number of web pages. Rate words or phrases as related if they appear more often near each other (e.g. within a couple of paragraphs) than randomly. Given two or more words or phrases, rate other words or phrases as interesting if they're related to more than one of the other words or phrases. A score for how interesting could be calculated. Print out the most interesting ones.
Yes. Do you know a general term for this so I could search for appropriate c# snippets? I would do it the way you suggested, this was my original idea. But programmatically how would you go about it?
First separate all words by space and/or punctuation. This leads to an array. Then taking the first and second entry, saving them somewhere. Then looking through the whole array to find the same two words in this order. Then assign a number of occurences to it in a new associative array. Then a synonyms-dictionary has to come into play. What was our goal again? I feel we are losing track here.

So, my approach (which need not be the same as your approach) would be
to look over the material regularly as Ternouth suggests, perhaps varying it
somehow -- shuffling the order, rewriting it in different words, etc. --
and wait for ideas to come bubbling up out of my subconscious.
Writing down your dreams can help, too. And solving cryptic crosswords.
Which question are you answering here?

I hope you find a computer program that does what you're talking about.
Let me know if you do, because I'm interested in that kind of thing.
I appreciate your interest. As mentioned above this is the route I want to go to.

PS:
1. Can you elaborate the connection between my task and Hypothes.is?
2. You seem competent, can you exert your wisdom in your day to day job?
 

Oogiem

Registered
cwoodgold;104193 said:
I hope you find a computer program that does what you're talking about.
Let me know if you do, because I'm interested in that kind of thing.

Two suggestions:

Look at how big companies like Amazon and Netflix program the see also or you might like parts of their search. I know when I buy something at Amazon I will get a list of related and possibly interesting things to look at. Some are related via words I use some by use. Netflix is getting better and better about accurately guessing how we will rate any given movie based on our ratings from the past.

Both I believe are using some form of heuristic programming software to do that.

I haven't loked at heuristic programming techniques in decades but I would expect that it has gone a bit further than when I last looked. Might be the search term to get you there...
 

cwoodgold

Registered
Tanz der Nacht;104197 said:
Danke.

Which question are you answering here?
I was answering this: "What could a search system look like that finds links between thoughts ...
How would you approach this task?"

In other words, I would probably approach the task using the human brain rather than
a computer program. However, I'm also interested in computer programs to do similar things, and perhaps in writing such programs myself.

1. Can you elaborate the connection between my task and Hypothes.is?
There isn't much connection. Both involve computer programs that process large amounts of data and that work with interconnections among that data. Both are computer programs I feel excited about.

2. You seem competent, can you exert your wisdom in your day to day job?
Thank you for the compliment, and thank you very much for asking the question.
Yes, sometimes. I also find interconnections among large amounts of data in my work; for example here http://bssa.geoscienceworld.org/content/89/1/165.short
 

cwoodgold

Registered
This page about statistically significant phrases may be quite relevant: http://alias-i.com/lingpipe/demos/tutorial/interestingPhrases/read-me.html

Your question to me helps me increase my focus on how to arrange things so I can do more of the type of thing that's really satisfying in the long run: higher horizons-of-focus stuff. That's why I really appreciated you asking.

I also like Mark Jantzen's signature question: "How did you capture that thought?" I'll sometimes get things done immediately so I don't have to bother putting them into any system, for example. That's one way of capturing a thought, and that question can help remind me to do stuff like that.
 

cwoodgold

Registered
Here's a computer method:

Step 1.
Someone analyses a lot of text (e.g. a huge number of web pages and/or books)
and publishes a data file like a dictionary. The entry for "orchid" might list about
a hundred or more related words and phrases, with a score beside each one to
show how closely connected it is. Maybe there's something like that out there
already.

Step 2.
Someone writes a program that uses that list along with their own private input.
It starts with words from their own input, finds related words using the list,
and maybe does more steps, finding words related to those words and so on.
At each step it can choose the highest-scoring words or phrases, or choose
words and phrases at random with higher-scoring ones more likely to be chosen.

At the second step it might assign negative weights to the original words (and so on).
That's counter-intuitive but I think it's similar to how the human brain works and may
be worth a try.

Here's a different method, using the human brain:

Select two phrases from your input, at random. Think up an idea that
is connected to both of them. Repeat this exercise many times over a number
of months. At first it may seem mechanistic, but eventually it gets you
thinking in a different way (the way solving cryptic crosswords does),
as well as increasing the salience of the input. Later, interesting ideas
may bubble up from the subconscious at odd times.

If you have difficulty thinking up an idea, lower your standards:
the idea is allowed to be silly and useless and is allowed to be only
loosely connected to the input. Lowering the standards in this way
is an important step in creative thinking. Sometimes a silly idea
turns out to actually be useful, or else to lead to another idea that's
useful.
 

Todd V

Registered
Re: finding synonymic words

The guys over at Nisus are working on a sweet new search product that does synonymic searches, similar to what you're looking for. Should release a product in early 2013. I forget the name of it, but it looks really good. I found out about it from their latest newsletter.
 

chrisstribbs

Registered
Hi!

So I am using a digital voice recorder to record anything I see important, ranging from business to private, from rants to new business ideas.

Every finalized idea is one wma-file.

I wrote a program to sort the wma-files into folders. From time to time I listen to the wma-files, convert them to text(manually) and insert them into a mindmap with mindmanager, which I sort hierarchically by area and type in turn.

This works very well, no idea is being lost, when I am out of ideas for a special topic, I listen to what I said and can get started again.

What could a search system look like that finds links between thoughts(written in the mindmap and in the wma files) or in general gives me good search results even when the keyword I searched for is not present but a synonym of it or related topic(for instance flower should output entries containing orchid aswell, even if they contain orchid but not the very keyword flower).

I prefer something ready-made but small adjustments to a given system are fine aswell.

How would you approach this task?
I would simply use Dragon to transcribe the voice files.
 
Top