– by Niels Michaeli Lauritsen (nlaur16) and Sofie Bundgaard Jørgensen (sojoe16)

Video of prototype: Click on: videoOfPrototype to watch the video

BitBucket-link: https://bitbucket.org/nlaur16/spotthis/src/master/

Introduction.

As a part of the course iOS programming an application was to be made with different constraints. The application should use one to more sensors from the phone or iPad. Furthermore, it should have specific elements regarding the view as for storage and graphics.

The project should as well as fulfill the requirements be a small journey into the world of iOS. The goal for the group was to learn the Swift language in general and how to build an app from scratch.

Basic idea.
Let’s say that you wish to sort your photografies by categories from your photo library.

You don’t want to do it manually because of the amount of time it will take. And you don’t want your new photo library app to be unintuitive or complicated. It should look like your normal photo library.

This was the “problem” that made us come up with the idea of creating SpotThis.

SpotThis is a simple application where you are able to take a picture with the camera within the app and have your existing photo library sorted by categories. Furthermore the app is intuitive with categories at your frontpage and from there you can easily navigate through your pictures.

The subject is important as it is often difficult to find your most important memories. Your memories is part of your life and will define a somewhat future you. We want to create an app that can help people organize their favourite pictures but also by a memory.

 

Methods and material.

The app is build in XCode and Swift 4 and the sensor used is the camera.

Methods.
First step of the process of creating the project is finding an idea. Brainstorming. The initial idea was some sort of a game or the category app. However since there wasn’t a concrete idea to the game, we went with the category app as this was the idea that we knew how to approach with regard to image recognition.

The brainstorming led to the conceptual design which consisted of two different designs with the same scope of features.

The first one looked like this:

The second looked like this:

Both conceptual designs were created in Balsamiq. Balsamiq is a computer software which allows for creating of prototypes with navigation between different views.

The overall thought with the app was that it should be intuitive and easy to navigate in. This was done in terms of easy access to folders/categories and quick access to deleting, favoriting and mapping the picture.

From the initial thought to the end result of the conceptual design phase the app added on buttons for seeing latest picture taken, and small thoughts about the overall feel and look of the app.

Prototype.
All projects go through a rough path of bad decisions and a lot of different types of idea and tries. This happened within the prototype phase. A lot of different approaches were tried out before the first simple prototype worked. This resulted in a long, ongoing, non giving and very confusing time period where there were no results.

The first prototype contained a collection view cell and a camera option which worked. It was also capable of analyzing a image using the cognitive services API from Microsoft. But in the prototype, we just analyzed a test image from a url. weren’t able of passing the image from the camera to the SDK yet. But it worked out fine, and we were able to pull out the informations we got from the response. So from the analyzed image we could get informations that we should use later on, such as tags, description and probabilities of objects on the image.

Evaluation of prototype
Our prototype was capable of analyzing a image and taking a picture with the camera. But it hadn’t the capability to combine theses two features. But implementing and making changes to the SDK which was not fully completed, was the biggest task when making the prototype. First we had problems being denied by the API because we was using the wrong URL to call the API, but as this wasn’t described in the documentation of the SDK we didn’t know that we should change it to get the API calls working. We also realised that the SDK did not support the tags, that was paired with a probability that the specific tag/object was in the picture. As this was a functionality we based our project on, we needed to implement this before we went further with the prototype.

Use-Case diagrams

Finding requirements
Based  on our first presentation where we only represented the basic idea and the mock-up of our app. We came up with theses requirements, based on the feedback we got, and the base project description.

Based on our first prototype we still need to fulfill a lot of theses requirements.

  • Functional requirements
    • A user should be able to save a picture
    • A user should be able to take a new picture
    • A user should be able to browser through categories
    • A user should be able to see pictures inside categories
    • A user should be able to choose image from the gallery
  • Non-Functional requirements
    • The app should be able to return the analyze of a picture within 5 seconds
    • The app should be able to put the picture into categories
    • The app should be able to analyze a picture from the gallery
    • The app should be able to save categories from past analyzed pictures

 

Implementation

The implementation section will describe which frameworks have been used (if any), classes and hereby how the model-view-controller pattern has been used, and which properties and methods of notice there is.

The SDK is created by Vladimir Danila and is used within the project to analyse images. See this link for the github repository.

MVC-model

This MVC model shows how the front page of our app should use our model(ImageInfo) to get some data that can be displayed in our view. The ViewController will in the didAppear() method refresh images every time the view appears again.

   override func viewDidAppear(_ animated: Bool) {
        let db = ImageInfoDB()
        imagesFromDb = db.loadImages()
        collectionView.reloadData()
        
    }

As seen above, an instance of our ImageInfoDB is created which is responsible for saving and loading ImageInfo’s from the core data. We then fetch all the images from the db and sets them to an imageFromDb variable. We then reload the data in the collectionview which is the “imagesFromDb” variable, so if any new images since last time has appeared in core data they will now be loaded.

   func loadImages() -> [ImageInfo]{
        let imageInfoFetch = NSFetchRequest<NSFetchRequestResult>
           (entityName: "ImageInfo")
        let imageinfos = try! managedContext.fetch(imageInfoFetch)
        return imageinfos as! [ImageInfo]
    }

But if you are looking at the MVC model and the code, you can see that the ImageInfo doesn’t notify the viewcontroller when there is changes. So we are not following mvc model correctly, but as time is short and we had several issues down the way this wasn’t our first priority.

Core-data is used to save our information about the analyzed images. We have an entity called ImageInfo which has three properties (String called ImagePath, String array called tags and a another String called descriptionText). In our gallery, every cell has a reference to a specific ImageInfo, and the cell also contains a UIImageview. Which means, when displaying an image in our cell we parse the image name to the method fetchImage(imagename) which returns an UIImage holding the image. We then put the image into our cell’s imageview.

The collectionView(cellForItemIt) is the method that populates all the cells with the images saved in document directory, and the name of the image is saved in the core-data entities.

    func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
        let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "Cell", for: indexPath) as! CollectionViewCell
        if(imagesFromDb.count > 0){
            let imageInfo = imagesFromDb[indexPath.item]
            let imageForCell = fetchImage(nameImage: imageInfo.imagePath!)
            
            if (imageInfo.tags.count > 0) {
                 cell.label.text = imageInfo.tags[0] as String
            } else {
                print("imageinfo tags are nil")
            }
            
            cell.imageViewInCell.image = imageForCell
            cell.configure(imageInfo: imageInfo)
            print(imageInfo.imagePath! + " was inserted into collection view")
            
        }
        return cell
    }

The saveImage method is used to save the information about a picture which contains (imagepath, image tags and image description). As seen in the code, an object is created of the entity imageInfo and set it’s variables to the data recived from the camera controller. And then tries to save the entity in the core-data context, and if that doesn’t succeds an error is printed.

func saveImage(imagePath: String, tags: [String], description: String){
        
        let imageInfoEntity = NSEntityDescription.entity(forEntityName: "ImageInfo", in: managedContext)
        
        let imageInfo = NSManagedObject(entity: imageInfoEntity!, insertInto: managedContext)
        
        imageInfo.setValue(imagePath, forKey: "imagePath")
        imageInfo.setValue(tags, forKey: "tags")
        imageInfo.setValue(description, forKey: "descriptionText")
        
        do {
            try managedContext.save()
        } catch let error as NSError {
            print("Could not save. \(error)")
        }
        
    }


Discussion

Throughout the process of creating the application it has been a key objective to integrate Microsoft Cognitive Services within the application. This was chosen since the gallery app should be able to distinguish photos from one another and then sort them by tags.

However, the app doesn’t meet as many requirements as it was expected too due to unforeseen complications. Troubles with Auto Layout, Core Data, SDK and Swift/Xcode in general have been setbacks during the project’s duration. Furthermore limited time have been an obstacle.

Evaluation from users have been that the application represents a good proof of concept, but is lacking a UI which is easy and intuitive to navigate through. A lot of the wanted and expected functionality has been implemented in the application, though it needs tweaks here and there to fully meet our requirements. Since the application isn’t implemented as fully as expected, evaluations have though been lacking as the application haven’t got the expected UI.

Good things and bad thing with our application
Some of the goods things about our application is that it’s capable of analyzing a image through Microsoft cognitive services API. The API will return a description of the image, and different tags that is related to the content of the image. After an image is analyzed it’s put into the database, which saves every image and it’s related tags and description. You can then on the frontpage of the app se all the images, and when you click on them you are getting redirected to another view that shows the tags and description.

Bad things about our app when take an image there is a delay before the image is analyzed because we have to wait for the API to analyze the image and send a response back.

The application is not capable of sorting the different tags, so you cannot select a specific tag and only see the images that contain that tag. Rotation of images and auto-layout on different devices are not fully supported. And the application has only been tested on an Ipad, so there might be issues on smaller IOS devices.

Things that still needs to be implemented:
As described the app does not meet our set requirements for the app, which results in a bit of a long list of features that still needs to be implemented.

  • Automatically categorize new pictures
  • Arrange categories in a shared view
  • Be able to favor pictures
  • See where the picture was taken on a map
  • Import pictures and have the app categorize them
  • Delete pictures
  • Prettier UI
  • Automatically adjusting cell size in UICollectionView to fit a picture

 

Conclusion

Did we succeed with our product?

No is the short answer. Too many of the requirements haven’t been met and to many complications during the development of the application have created to many setbacks.

The app however does fulfill the requirement with the highest priority within the group; use of Microsoft Cognitive Services. The idea/app is centered around the concept of analysing images, which is what cognitive services provide. That said, the product itself is missing some of the most basic features to be able to compete with other gallery apps. The folder structure, correct rotation of pictures in collectionview etc.

Swift as a programming language and XCode as an IDE, have too heard a fair share of curse words. Swift isn’t that hard to learn, but there is still some difference and other concepts compared to languages like C# and Java .

XCode too registers errors in the code much later than IntelliJ as an IDE, making the development slower.

We hoped to have gotten much further in the development of the app, but problems with the Swift SDK slowed us down. Hereby said that we didn’t manage to incorporate as many features as we would like. And based on the problems in regards to the features, we decided not to focus on the design only some autolayout to fix the worst parts.

 

Perspective

For further development the app should be able to categories the pictures taken within the app and sort them accordingly to other pictures. Furthermore the MVC-pattern should be implemented correctly. A user should to be able to favor pictures which then is placed in a seperate folder. It too would be preferable if the user could see on a map, where the picture was taken. The list shows many options for further development, where some hopefully will be implemented before the exam in January.

Leave a Reply