I got this to work at 5.3.2021, and from the start it was really cool to play with. In the image above, you can see how visually similar images have been loaded into the scene.
It's been some time and I can't remember it, and mainly my workflow was a lot of trial and error, very messy and barely working. But, I was interested in making it work, and I did it. It was a really cool feeling.
First, I got the features of each image. https://www.tensorflow.org/tutorials/images/transfer_learning_with_hub?hl=cs had this tutorial for extracting the probability of labels, and I got the vector of features for each image. The columns here are the different features, and there's one row for each image. The .csv file is around 37mb and there's close to 2000 rows. I had to batch the processing, since I had so many images. The example code I was modifying for my needs was saving the rows into a Pandas dataframe, I remember I had some problems with getting the output into .csv structured like I wanted.
Then, you'd calculate the cosine similarity for the images. I saw this done on another example, and with some edits, managed to get the distance of each image to each other. I'm so happy I didn't have to implement it. The main challenge was to get the format right so I could use scikit's cosine_similarity function.
This resulting nice .csv file shows the similarity score between any of my images. It can be then sorted to see the closest images.
At the moment, I just load the .csv files in Unity. It's not pretty, but currently it's working pretty well - my reasoning is that there's lots of other things to do which are a better use for my time.
In the videos below, the most similar images are loaded when I click any image.
Not only do I load the .csv with the cosine distances, I also load the file with all the image feature vectors too. I wanted to create a scoring system that would calculate the average distance of a group of images. I think I had this implemented already at one point, but the values it would return didn't seem intuitively correct. I might revisit this at some point to see what can be done.
Here's a delightful note I made 11. October 2021 of how I planned to calculate a score for a group of items:
